2023-02-13 12:21:57,404 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683 2023-02-13 12:21:57,415 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-13 12:21:57,426 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-13 12:21:57,426 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836, deleteOnExit=true 2023-02-13 12:21:57,427 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-13 12:21:57,427 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/test.cache.data in system properties and HBase conf 2023-02-13 12:21:57,427 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/hadoop.tmp.dir in system properties and HBase conf 2023-02-13 12:21:57,428 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/hadoop.log.dir in system properties and HBase conf 2023-02-13 12:21:57,428 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-13 12:21:57,429 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-13 12:21:57,429 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-13 12:21:57,526 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-13 12:21:57,842 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-13 12:21:57,846 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-13 12:21:57,846 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-13 12:21:57,846 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-13 12:21:57,847 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 12:21:57,847 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-13 12:21:57,847 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-13 12:21:57,847 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 12:21:57,848 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 12:21:57,848 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-13 12:21:57,848 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/nfs.dump.dir in system properties and HBase conf 2023-02-13 12:21:57,849 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/java.io.tmpdir in system properties and HBase conf 2023-02-13 12:21:57,849 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 12:21:57,849 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-13 12:21:57,849 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-13 12:21:58,254 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 12:21:58,257 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 12:21:58,636 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-13 12:21:58,779 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-13 12:21:58,790 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 12:21:58,820 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 12:21:58,845 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/java.io.tmpdir/Jetty_localhost_localdomain_41897_hdfs____.1rhr3h/webapp 2023-02-13 12:21:58,946 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:41897 2023-02-13 12:21:58,986 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 12:21:58,987 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 12:21:59,482 WARN [Listener at localhost.localdomain/43631] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 12:21:59,565 WARN [Listener at localhost.localdomain/43631] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 12:21:59,584 WARN [Listener at localhost.localdomain/43631] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 12:21:59,589 INFO [Listener at localhost.localdomain/43631] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 12:21:59,593 INFO [Listener at localhost.localdomain/43631] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/java.io.tmpdir/Jetty_localhost_33513_datanode____.3uy95u/webapp 2023-02-13 12:21:59,668 INFO [Listener at localhost.localdomain/43631] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33513 2023-02-13 12:21:59,923 WARN [Listener at localhost.localdomain/44837] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 12:21:59,932 WARN [Listener at localhost.localdomain/44837] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 12:21:59,936 WARN [Listener at localhost.localdomain/44837] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 12:21:59,938 INFO [Listener at localhost.localdomain/44837] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 12:21:59,942 INFO [Listener at localhost.localdomain/44837] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/java.io.tmpdir/Jetty_localhost_45579_datanode____.6m48kj/webapp 2023-02-13 12:22:00,020 INFO [Listener at localhost.localdomain/44837] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45579 2023-02-13 12:22:00,029 WARN [Listener at localhost.localdomain/44543] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 12:22:00,044 WARN [Listener at localhost.localdomain/44543] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 12:22:00,048 WARN [Listener at localhost.localdomain/44543] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 12:22:00,049 INFO [Listener at localhost.localdomain/44543] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 12:22:00,055 INFO [Listener at localhost.localdomain/44543] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/java.io.tmpdir/Jetty_localhost_43333_datanode____mys1cr/webapp 2023-02-13 12:22:00,130 INFO [Listener at localhost.localdomain/44543] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43333 2023-02-13 12:22:00,139 WARN [Listener at localhost.localdomain/46179] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 12:22:01,101 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe66bdb97fb109be: Processing first storage report for DS-c60bd1cd-ef15-4483-9b0e-1e3428eeb8f8 from datanode 5b5fd4b5-8f31-4d5e-9179-89c46b38b1c7 2023-02-13 12:22:01,103 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe66bdb97fb109be: from storage DS-c60bd1cd-ef15-4483-9b0e-1e3428eeb8f8 node DatanodeRegistration(127.0.0.1:39855, datanodeUuid=5b5fd4b5-8f31-4d5e-9179-89c46b38b1c7, infoPort=41673, infoSecurePort=0, ipcPort=46179, storageInfo=lv=-57;cid=testClusterID;nsid=7529154;c=1676290918312), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-13 12:22:01,103 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4d74ca68c44a99fe: Processing first storage report for DS-618d2db0-bb39-4e38-8988-eb9c8891f87e from datanode b1c51ee3-df0b-49ef-bd7b-a0c7d8646013 2023-02-13 12:22:01,103 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4d74ca68c44a99fe: from storage DS-618d2db0-bb39-4e38-8988-eb9c8891f87e node DatanodeRegistration(127.0.0.1:36969, datanodeUuid=b1c51ee3-df0b-49ef-bd7b-a0c7d8646013, infoPort=36595, infoSecurePort=0, ipcPort=44543, storageInfo=lv=-57;cid=testClusterID;nsid=7529154;c=1676290918312), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 12:22:01,103 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe573c701c715cb45: Processing first storage report for DS-bbcaef41-1b8b-4853-a86f-103f917b438e from datanode 01bf2301-7426-448b-8288-7b5ccc31096b 2023-02-13 12:22:01,103 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe573c701c715cb45: from storage DS-bbcaef41-1b8b-4853-a86f-103f917b438e node DatanodeRegistration(127.0.0.1:39681, datanodeUuid=01bf2301-7426-448b-8288-7b5ccc31096b, infoPort=43365, infoSecurePort=0, ipcPort=44837, storageInfo=lv=-57;cid=testClusterID;nsid=7529154;c=1676290918312), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 12:22:01,103 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe66bdb97fb109be: Processing first storage report for DS-4ff12821-66b3-4714-b3ec-195a2911a5bb from datanode 5b5fd4b5-8f31-4d5e-9179-89c46b38b1c7 2023-02-13 12:22:01,103 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe66bdb97fb109be: from storage DS-4ff12821-66b3-4714-b3ec-195a2911a5bb node DatanodeRegistration(127.0.0.1:39855, datanodeUuid=5b5fd4b5-8f31-4d5e-9179-89c46b38b1c7, infoPort=41673, infoSecurePort=0, ipcPort=46179, storageInfo=lv=-57;cid=testClusterID;nsid=7529154;c=1676290918312), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 12:22:01,103 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4d74ca68c44a99fe: Processing first storage report for DS-4fcff5d5-e9f9-4d3e-af10-80e2ac190a97 from datanode b1c51ee3-df0b-49ef-bd7b-a0c7d8646013 2023-02-13 12:22:01,104 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4d74ca68c44a99fe: from storage DS-4fcff5d5-e9f9-4d3e-af10-80e2ac190a97 node DatanodeRegistration(127.0.0.1:36969, datanodeUuid=b1c51ee3-df0b-49ef-bd7b-a0c7d8646013, infoPort=36595, infoSecurePort=0, ipcPort=44543, storageInfo=lv=-57;cid=testClusterID;nsid=7529154;c=1676290918312), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 12:22:01,104 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe573c701c715cb45: Processing first storage report for DS-ccb23efb-1d61-498a-962d-e9a7c408bd81 from datanode 01bf2301-7426-448b-8288-7b5ccc31096b 2023-02-13 12:22:01,104 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe573c701c715cb45: from storage DS-ccb23efb-1d61-498a-962d-e9a7c408bd81 node DatanodeRegistration(127.0.0.1:39681, datanodeUuid=01bf2301-7426-448b-8288-7b5ccc31096b, infoPort=43365, infoSecurePort=0, ipcPort=44837, storageInfo=lv=-57;cid=testClusterID;nsid=7529154;c=1676290918312), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 12:22:01,123 DEBUG [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683 2023-02-13 12:22:01,181 INFO [Listener at localhost.localdomain/46179] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/zookeeper_0, clientPort=62541, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-13 12:22:01,193 INFO [Listener at localhost.localdomain/46179] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=62541 2023-02-13 12:22:01,199 INFO [Listener at localhost.localdomain/46179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 12:22:01,201 INFO [Listener at localhost.localdomain/46179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 12:22:01,831 INFO [Listener at localhost.localdomain/46179] util.FSUtils(479): Created version file at hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83 with version=8 2023-02-13 12:22:01,831 INFO [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/hbase-staging 2023-02-13 12:22:02,102 INFO [Listener at localhost.localdomain/46179] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-13 12:22:02,477 INFO [Listener at localhost.localdomain/46179] client.ConnectionUtils(127): master/jenkins-hbase16:0 server-side Connection retries=18 2023-02-13 12:22:02,501 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 12:22:02,502 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 12:22:02,502 INFO [Listener at localhost.localdomain/46179] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 12:22:02,502 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 12:22:02,502 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 12:22:02,615 INFO [Listener at localhost.localdomain/46179] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 12:22:02,673 DEBUG [Listener at localhost.localdomain/46179] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-13 12:22:02,751 INFO [Listener at localhost.localdomain/46179] ipc.NettyRpcServer(120): Bind to /188.40.62.62:46677 2023-02-13 12:22:02,759 INFO [Listener at localhost.localdomain/46179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 12:22:02,761 INFO [Listener at localhost.localdomain/46179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 12:22:02,778 INFO [Listener at localhost.localdomain/46179] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46677 connecting to ZooKeeper ensemble=127.0.0.1:62541 2023-02-13 12:22:02,880 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:466770x0, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 12:22:02,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:46677-0x10052d701450000 connected 2023-02-13 12:22:02,982 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ZKUtil(164): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 12:22:02,984 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ZKUtil(164): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 12:22:02,992 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ZKUtil(164): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 12:22:02,998 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46677 2023-02-13 12:22:02,999 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46677 2023-02-13 12:22:02,999 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46677 2023-02-13 12:22:03,000 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46677 2023-02-13 12:22:03,000 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46677 2023-02-13 12:22:03,005 INFO [Listener at localhost.localdomain/46179] master.HMaster(439): hbase.rootdir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83, hbase.cluster.distributed=false 2023-02-13 12:22:03,061 INFO [Listener at localhost.localdomain/46179] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=18 2023-02-13 12:22:03,062 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 12:22:03,062 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 12:22:03,062 INFO [Listener at localhost.localdomain/46179] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 12:22:03,062 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 12:22:03,062 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 12:22:03,066 INFO [Listener at localhost.localdomain/46179] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 12:22:03,069 INFO [Listener at localhost.localdomain/46179] ipc.NettyRpcServer(120): Bind to /188.40.62.62:34139 2023-02-13 12:22:03,071 INFO [Listener at localhost.localdomain/46179] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 12:22:03,076 INFO [Listener at localhost.localdomain/46179] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 12:22:03,077 INFO [Listener at localhost.localdomain/46179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 12:22:03,078 INFO [Listener at localhost.localdomain/46179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 12:22:03,080 INFO [Listener at localhost.localdomain/46179] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34139 connecting to ZooKeeper ensemble=127.0.0.1:62541 2023-02-13 12:22:03,091 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:341390x0, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 12:22:03,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34139-0x10052d701450001 connected 2023-02-13 12:22:03,092 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ZKUtil(164): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 12:22:03,093 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ZKUtil(164): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 12:22:03,094 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ZKUtil(164): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 12:22:03,095 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34139 2023-02-13 12:22:03,095 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34139 2023-02-13 12:22:03,096 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34139 2023-02-13 12:22:03,097 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34139 2023-02-13 12:22:03,097 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34139 2023-02-13 12:22:03,112 INFO [Listener at localhost.localdomain/46179] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=18 2023-02-13 12:22:03,112 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 12:22:03,112 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 12:22:03,113 INFO [Listener at localhost.localdomain/46179] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 12:22:03,113 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 12:22:03,113 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 12:22:03,113 INFO [Listener at localhost.localdomain/46179] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 12:22:03,115 INFO [Listener at localhost.localdomain/46179] ipc.NettyRpcServer(120): Bind to /188.40.62.62:34079 2023-02-13 12:22:03,115 INFO [Listener at localhost.localdomain/46179] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 12:22:03,117 INFO [Listener at localhost.localdomain/46179] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 12:22:03,117 INFO [Listener at localhost.localdomain/46179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 12:22:03,119 INFO [Listener at localhost.localdomain/46179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 12:22:03,120 INFO [Listener at localhost.localdomain/46179] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34079 connecting to ZooKeeper ensemble=127.0.0.1:62541 2023-02-13 12:22:03,133 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:340790x0, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 12:22:03,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34079-0x10052d701450002 connected 2023-02-13 12:22:03,134 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ZKUtil(164): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 12:22:03,135 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ZKUtil(164): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 12:22:03,136 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ZKUtil(164): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 12:22:03,136 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34079 2023-02-13 12:22:03,136 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34079 2023-02-13 12:22:03,137 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34079 2023-02-13 12:22:03,137 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34079 2023-02-13 12:22:03,138 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34079 2023-02-13 12:22:03,149 INFO [Listener at localhost.localdomain/46179] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=18 2023-02-13 12:22:03,149 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 12:22:03,149 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 12:22:03,149 INFO [Listener at localhost.localdomain/46179] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 12:22:03,149 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 12:22:03,149 INFO [Listener at localhost.localdomain/46179] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 12:22:03,149 INFO [Listener at localhost.localdomain/46179] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 12:22:03,151 INFO [Listener at localhost.localdomain/46179] ipc.NettyRpcServer(120): Bind to /188.40.62.62:36049 2023-02-13 12:22:03,151 INFO [Listener at localhost.localdomain/46179] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 12:22:03,152 INFO [Listener at localhost.localdomain/46179] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 12:22:03,153 INFO [Listener at localhost.localdomain/46179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 12:22:03,155 INFO [Listener at localhost.localdomain/46179] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 12:22:03,157 INFO [Listener at localhost.localdomain/46179] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36049 connecting to ZooKeeper ensemble=127.0.0.1:62541 2023-02-13 12:22:03,170 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:360490x0, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 12:22:03,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36049-0x10052d701450003 connected 2023-02-13 12:22:03,171 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ZKUtil(164): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 12:22:03,172 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ZKUtil(164): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 12:22:03,173 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ZKUtil(164): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 12:22:03,173 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36049 2023-02-13 12:22:03,173 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36049 2023-02-13 12:22:03,174 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36049 2023-02-13 12:22:03,174 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36049 2023-02-13 12:22:03,174 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36049 2023-02-13 12:22:03,176 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase16.apache.org,46677,1676290921946 2023-02-13 12:22:03,192 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 12:22:03,193 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase16.apache.org,46677,1676290921946 2023-02-13 12:22:03,220 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 12:22:03,220 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 12:22:03,220 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 12:22:03,220 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 12:22:03,220 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 12:22:03,221 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 12:22:03,223 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase16.apache.org,46677,1676290921946 from backup master directory 2023-02-13 12:22:03,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 12:22:03,233 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase16.apache.org,46677,1676290921946 2023-02-13 12:22:03,233 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 12:22:03,234 WARN [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 12:22:03,234 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase16.apache.org,46677,1676290921946 2023-02-13 12:22:03,237 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-13 12:22:03,239 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-13 12:22:03,333 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/hbase.id with ID: f991c098-b383-4f7f-bd80-55c648ab3d6d 2023-02-13 12:22:03,378 INFO [master/jenkins-hbase16:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 12:22:03,403 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 12:22:03,451 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x006d038c to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:03,485 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ea77cc0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:03,503 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 12:22:03,504 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 12:22:03,518 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-13 12:22:03,518 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-13 12:22:03,519 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 12:22:03,523 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 12:22:03,524 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 12:22:03,557 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/MasterData/data/master/store-tmp 2023-02-13 12:22:03,591 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:03,592 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 12:22:03,592 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 12:22:03,592 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 12:22:03,592 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 12:22:03,592 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 12:22:03,593 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 12:22:03,593 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676290923591Disabling compacts and flushes for region at 1676290923592 (+1 ms)Disabling writes for close at 1676290923592Writing region close event to WAL at 1676290923593 (+1 ms)Closed at 1676290923593 2023-02-13 12:22:03,595 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/MasterData/WALs/jenkins-hbase16.apache.org,46677,1676290921946 2023-02-13 12:22:03,614 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C46677%2C1676290921946, suffix=, logDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/MasterData/WALs/jenkins-hbase16.apache.org,46677,1676290921946, archiveDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/MasterData/oldWALs, maxLogs=10 2023-02-13 12:22:03,664 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39681,DS-bbcaef41-1b8b-4853-a86f-103f917b438e,DISK] 2023-02-13 12:22:03,664 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39855,DS-c60bd1cd-ef15-4483-9b0e-1e3428eeb8f8,DISK] 2023-02-13 12:22:03,664 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36969,DS-618d2db0-bb39-4e38-8988-eb9c8891f87e,DISK] 2023-02-13 12:22:03,672 DEBUG [RS-EventLoopGroup-5-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 12:22:03,739 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/MasterData/WALs/jenkins-hbase16.apache.org,46677,1676290921946/jenkins-hbase16.apache.org%2C46677%2C1676290921946.1676290923624 2023-02-13 12:22:03,739 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36969,DS-618d2db0-bb39-4e38-8988-eb9c8891f87e,DISK], DatanodeInfoWithStorage[127.0.0.1:39681,DS-bbcaef41-1b8b-4853-a86f-103f917b438e,DISK], DatanodeInfoWithStorage[127.0.0.1:39855,DS-c60bd1cd-ef15-4483-9b0e-1e3428eeb8f8,DISK]] 2023-02-13 12:22:03,740 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 12:22:03,741 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:03,744 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 12:22:03,745 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 12:22:03,795 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 12:22:03,802 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 12:22:03,828 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 12:22:03,839 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:03,844 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 12:22:03,846 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 12:22:03,860 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 12:22:03,864 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:03,865 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71452343, jitterRate=0.06472288072109222}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:03,866 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676290923767Initializing all the Stores at 1676290923770 (+3 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676290923770Cleaning up temporary data from old regions at 1676290923852 (+82 ms)Cleaning up detritus from prior splits at 1676290923853 (+1 ms)Region opened successfully at 1676290923866 (+13 ms) 2023-02-13 12:22:03,867 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 12:22:03,884 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 12:22:03,884 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 12:22:03,886 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 12:22:03,887 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-13 12:22:03,912 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 24 msec 2023-02-13 12:22:03,912 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 12:22:03,932 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-13 12:22:03,937 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-13 12:22:03,957 INFO [master/jenkins-hbase16:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 12:22:03,960 INFO [master/jenkins-hbase16:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 12:22:03,961 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-13 12:22:03,965 INFO [master/jenkins-hbase16:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 12:22:03,968 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 12:22:04,011 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 12:22:04,013 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 12:22:04,014 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 12:22:04,025 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 12:22:04,036 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 12:22:04,036 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 12:22:04,036 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 12:22:04,036 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 12:22:04,037 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 12:22:04,037 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase16.apache.org,46677,1676290921946, sessionid=0x10052d701450000, setting cluster-up flag (Was=false) 2023-02-13 12:22:04,066 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 12:22:04,091 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 12:22:04,093 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase16.apache.org,46677,1676290921946 2023-02-13 12:22:04,111 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 12:22:04,136 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 12:22:04,138 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase16.apache.org,46677,1676290921946 2023-02-13 12:22:04,142 WARN [master/jenkins-hbase16:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp 2023-02-13 12:22:04,179 INFO [RS:0;jenkins-hbase16:34139] regionserver.HRegionServer(952): ClusterId : f991c098-b383-4f7f-bd80-55c648ab3d6d 2023-02-13 12:22:04,179 INFO [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(952): ClusterId : f991c098-b383-4f7f-bd80-55c648ab3d6d 2023-02-13 12:22:04,179 INFO [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(952): ClusterId : f991c098-b383-4f7f-bd80-55c648ab3d6d 2023-02-13 12:22:04,183 DEBUG [RS:2;jenkins-hbase16:36049] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 12:22:04,183 DEBUG [RS:1;jenkins-hbase16:34079] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 12:22:04,183 DEBUG [RS:0;jenkins-hbase16:34139] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 12:22:04,204 DEBUG [RS:2;jenkins-hbase16:36049] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 12:22:04,204 DEBUG [RS:1;jenkins-hbase16:34079] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 12:22:04,204 DEBUG [RS:0;jenkins-hbase16:34139] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 12:22:04,204 DEBUG [RS:1;jenkins-hbase16:34079] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 12:22:04,204 DEBUG [RS:2;jenkins-hbase16:36049] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 12:22:04,204 DEBUG [RS:0;jenkins-hbase16:34139] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 12:22:04,220 DEBUG [RS:2;jenkins-hbase16:36049] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 12:22:04,220 DEBUG [RS:1;jenkins-hbase16:34079] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 12:22:04,220 DEBUG [RS:0;jenkins-hbase16:34139] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 12:22:04,222 DEBUG [RS:2;jenkins-hbase16:36049] zookeeper.ReadOnlyZKClient(139): Connect 0x32c50248 to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:04,223 DEBUG [RS:0;jenkins-hbase16:34139] zookeeper.ReadOnlyZKClient(139): Connect 0x5460a4d4 to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:04,223 DEBUG [RS:1;jenkins-hbase16:34079] zookeeper.ReadOnlyZKClient(139): Connect 0x4d1e48bf to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:04,272 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-13 12:22:04,280 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-02-13 12:22:04,280 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-02-13 12:22:04,280 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-02-13 12:22:04,281 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-02-13 12:22:04,281 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase16:0, corePoolSize=10, maxPoolSize=10 2023-02-13 12:22:04,281 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,281 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-02-13 12:22:04,281 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,282 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676290954282 2023-02-13 12:22:04,283 DEBUG [RS:2;jenkins-hbase16:36049] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c02fa60, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:04,284 DEBUG [RS:1;jenkins-hbase16:34079] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43d9af6c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:04,284 DEBUG [RS:0;jenkins-hbase16:34139] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5aa75028, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:04,284 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 12:22:04,284 DEBUG [RS:2;jenkins-hbase16:36049] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@383d6938, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-02-13 12:22:04,284 DEBUG [RS:0;jenkins-hbase16:34139] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e98cfa4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-02-13 12:22:04,284 DEBUG [RS:1;jenkins-hbase16:34079] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47e623cf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-02-13 12:22:04,289 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 12:22:04,290 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-13 12:22:04,294 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 12:22:04,296 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 12:22:04,303 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 12:22:04,304 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 12:22:04,304 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 12:22:04,304 DEBUG [RS:2;jenkins-hbase16:36049] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase16:36049 2023-02-13 12:22:04,305 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 12:22:04,307 DEBUG [RS:0;jenkins-hbase16:34139] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase16:34139 2023-02-13 12:22:04,308 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,308 DEBUG [RS:1;jenkins-hbase16:34079] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase16:34079 2023-02-13 12:22:04,309 INFO [RS:2;jenkins-hbase16:36049] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 12:22:04,309 INFO [RS:1;jenkins-hbase16:34079] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 12:22:04,310 INFO [RS:1;jenkins-hbase16:34079] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 12:22:04,309 INFO [RS:0;jenkins-hbase16:34139] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 12:22:04,310 INFO [RS:0;jenkins-hbase16:34139] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 12:22:04,310 DEBUG [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 12:22:04,310 INFO [RS:2;jenkins-hbase16:36049] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 12:22:04,310 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 12:22:04,310 DEBUG [RS:0;jenkins-hbase16:34139] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 12:22:04,310 DEBUG [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 12:22:04,312 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 12:22:04,312 INFO [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase16.apache.org,46677,1676290921946 with isa=jenkins-hbase16.apache.org/188.40.62.62:34079, startcode=1676290923111 2023-02-13 12:22:04,312 INFO [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase16.apache.org,46677,1676290921946 with isa=jenkins-hbase16.apache.org/188.40.62.62:36049, startcode=1676290923148 2023-02-13 12:22:04,312 INFO [RS:0;jenkins-hbase16:34139] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase16.apache.org,46677,1676290921946 with isa=jenkins-hbase16.apache.org/188.40.62.62:34139, startcode=1676290923061 2023-02-13 12:22:04,312 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 12:22:04,316 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 12:22:04,317 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 12:22:04,320 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1676290924320,5,FailOnTimeoutGroup] 2023-02-13 12:22:04,321 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1676290924320,5,FailOnTimeoutGroup] 2023-02-13 12:22:04,321 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,321 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 12:22:04,322 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,322 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,340 DEBUG [RS:2;jenkins-hbase16:36049] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 12:22:04,340 DEBUG [RS:0;jenkins-hbase16:34139] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 12:22:04,340 DEBUG [RS:1;jenkins-hbase16:34079] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 12:22:04,343 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 12:22:04,344 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 12:22:04,344 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83 2023-02-13 12:22:04,371 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:04,375 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 12:22:04,381 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/info 2023-02-13 12:22:04,382 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 12:22:04,383 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:04,383 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 12:22:04,386 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/rep_barrier 2023-02-13 12:22:04,387 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 12:22:04,388 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:04,388 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 12:22:04,391 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/table 2023-02-13 12:22:04,391 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 12:22:04,392 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:04,395 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740 2023-02-13 12:22:04,396 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740 2023-02-13 12:22:04,400 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 12:22:04,402 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 12:22:04,405 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:04,406 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64488025, jitterRate=-0.0390535444021225}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 12:22:04,407 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676290924372Initializing all the Stores at 1676290924374 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676290924374Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676290924374Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676290924374Cleaning up temporary data from old regions at 1676290924398 (+24 ms)Cleaning up detritus from prior splits at 1676290924399 (+1 ms)Region opened successfully at 1676290924407 (+8 ms) 2023-02-13 12:22:04,407 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 12:22:04,407 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 12:22:04,407 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 12:22:04,407 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 12:22:04,407 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 12:22:04,408 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:40543, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 12:22:04,408 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:42507, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 12:22:04,408 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:54233, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 12:22:04,409 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 12:22:04,410 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676290924407Disabling compacts and flushes for region at 1676290924407Disabling writes for close at 1676290924407Writing region close event to WAL at 1676290924409 (+2 ms)Closed at 1676290924409 2023-02-13 12:22:04,415 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 12:22:04,415 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-13 12:22:04,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46677] master.ServerManager(394): Registering regionserver=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:04,421 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.ServerManager(394): Registering regionserver=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:04,422 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46677] master.ServerManager(394): Registering regionserver=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:04,427 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 12:22:04,439 DEBUG [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83 2023-02-13 12:22:04,439 DEBUG [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:43631 2023-02-13 12:22:04,440 DEBUG [RS:0;jenkins-hbase16:34139] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83 2023-02-13 12:22:04,440 DEBUG [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 12:22:04,440 DEBUG [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83 2023-02-13 12:22:04,440 DEBUG [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:43631 2023-02-13 12:22:04,440 DEBUG [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 12:22:04,440 DEBUG [RS:0;jenkins-hbase16:34139] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:43631 2023-02-13 12:22:04,440 DEBUG [RS:0;jenkins-hbase16:34139] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 12:22:04,444 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 12:22:04,447 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:04,483 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 12:22:04,495 DEBUG [RS:0;jenkins-hbase16:34139] zookeeper.ZKUtil(162): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:04,495 DEBUG [RS:2;jenkins-hbase16:36049] zookeeper.ZKUtil(162): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:04,495 WARN [RS:0;jenkins-hbase16:34139] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 12:22:04,495 WARN [RS:2;jenkins-hbase16:36049] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 12:22:04,496 DEBUG [RS:1;jenkins-hbase16:34079] zookeeper.ZKUtil(162): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:04,497 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,34079,1676290923111] 2023-02-13 12:22:04,497 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,36049,1676290923148] 2023-02-13 12:22:04,497 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,34139,1676290923061] 2023-02-13 12:22:04,496 INFO [RS:0;jenkins-hbase16:34139] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 12:22:04,496 INFO [RS:2;jenkins-hbase16:36049] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 12:22:04,497 DEBUG [RS:0;jenkins-hbase16:34139] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/WALs/jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:04,497 DEBUG [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/WALs/jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:04,497 WARN [RS:1;jenkins-hbase16:34079] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 12:22:04,498 INFO [RS:1;jenkins-hbase16:34079] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 12:22:04,498 DEBUG [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/WALs/jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:04,516 DEBUG [RS:2;jenkins-hbase16:36049] zookeeper.ZKUtil(162): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:04,516 DEBUG [RS:0;jenkins-hbase16:34139] zookeeper.ZKUtil(162): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:04,516 DEBUG [RS:1;jenkins-hbase16:34079] zookeeper.ZKUtil(162): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:04,516 DEBUG [RS:2;jenkins-hbase16:36049] zookeeper.ZKUtil(162): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:04,516 DEBUG [RS:0;jenkins-hbase16:34139] zookeeper.ZKUtil(162): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:04,517 DEBUG [RS:1;jenkins-hbase16:34079] zookeeper.ZKUtil(162): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:04,517 DEBUG [RS:2;jenkins-hbase16:36049] zookeeper.ZKUtil(162): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:04,517 DEBUG [RS:0;jenkins-hbase16:34139] zookeeper.ZKUtil(162): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:04,517 DEBUG [RS:1;jenkins-hbase16:34079] zookeeper.ZKUtil(162): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:04,527 DEBUG [RS:2;jenkins-hbase16:36049] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 12:22:04,527 DEBUG [RS:0;jenkins-hbase16:34139] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 12:22:04,527 DEBUG [RS:1;jenkins-hbase16:34079] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 12:22:04,535 INFO [RS:2;jenkins-hbase16:36049] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 12:22:04,536 INFO [RS:0;jenkins-hbase16:34139] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 12:22:04,535 INFO [RS:1;jenkins-hbase16:34079] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 12:22:04,553 INFO [RS:2;jenkins-hbase16:36049] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 12:22:04,553 INFO [RS:1;jenkins-hbase16:34079] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 12:22:04,553 INFO [RS:0;jenkins-hbase16:34139] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 12:22:04,557 INFO [RS:2;jenkins-hbase16:36049] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 12:22:04,557 INFO [RS:0;jenkins-hbase16:34139] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 12:22:04,557 INFO [RS:2;jenkins-hbase16:36049] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,557 INFO [RS:1;jenkins-hbase16:34079] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 12:22:04,557 INFO [RS:0;jenkins-hbase16:34139] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,558 INFO [RS:1;jenkins-hbase16:34079] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,559 INFO [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 12:22:04,559 INFO [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 12:22:04,559 INFO [RS:0;jenkins-hbase16:34139] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 12:22:04,567 INFO [RS:0;jenkins-hbase16:34139] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,567 INFO [RS:2;jenkins-hbase16:36049] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,567 INFO [RS:1;jenkins-hbase16:34079] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,567 DEBUG [RS:0;jenkins-hbase16:34139] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,567 DEBUG [RS:2;jenkins-hbase16:36049] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,567 DEBUG [RS:0;jenkins-hbase16:34139] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,567 DEBUG [RS:1;jenkins-hbase16:34079] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,567 DEBUG [RS:0;jenkins-hbase16:34139] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,567 DEBUG [RS:2;jenkins-hbase16:36049] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:0;jenkins-hbase16:34139] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:2;jenkins-hbase16:36049] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:0;jenkins-hbase16:34139] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,567 DEBUG [RS:1;jenkins-hbase16:34079] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:0;jenkins-hbase16:34139] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-02-13 12:22:04,568 DEBUG [RS:2;jenkins-hbase16:36049] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:0;jenkins-hbase16:34139] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:1;jenkins-hbase16:34079] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:2;jenkins-hbase16:36049] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:1;jenkins-hbase16:34079] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:2;jenkins-hbase16:36049] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-02-13 12:22:04,568 DEBUG [RS:1;jenkins-hbase16:34079] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:0;jenkins-hbase16:34139] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:1;jenkins-hbase16:34079] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-02-13 12:22:04,568 DEBUG [RS:2;jenkins-hbase16:36049] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:1;jenkins-hbase16:34079] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:0;jenkins-hbase16:34139] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:1;jenkins-hbase16:34079] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:2;jenkins-hbase16:36049] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:1;jenkins-hbase16:34079] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:0;jenkins-hbase16:34139] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,569 DEBUG [RS:1;jenkins-hbase16:34079] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,568 DEBUG [RS:2;jenkins-hbase16:36049] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,569 DEBUG [RS:2;jenkins-hbase16:36049] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-02-13 12:22:04,570 INFO [RS:1;jenkins-hbase16:34079] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,570 INFO [RS:0;jenkins-hbase16:34139] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,570 INFO [RS:1;jenkins-hbase16:34079] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,570 INFO [RS:0;jenkins-hbase16:34139] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,570 INFO [RS:2;jenkins-hbase16:36049] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,570 INFO [RS:0;jenkins-hbase16:34139] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,570 INFO [RS:2;jenkins-hbase16:36049] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,570 INFO [RS:1;jenkins-hbase16:34079] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,571 INFO [RS:2;jenkins-hbase16:36049] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,587 INFO [RS:1;jenkins-hbase16:34079] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 12:22:04,587 INFO [RS:2;jenkins-hbase16:36049] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 12:22:04,587 INFO [RS:0;jenkins-hbase16:34139] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 12:22:04,590 INFO [RS:2;jenkins-hbase16:36049] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,36049,1676290923148-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,590 INFO [RS:0;jenkins-hbase16:34139] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,34139,1676290923061-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,590 INFO [RS:1;jenkins-hbase16:34079] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,34079,1676290923111-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:04,600 DEBUG [jenkins-hbase16:46677] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-13 12:22:04,605 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 12:22:04,608 INFO [RS:2;jenkins-hbase16:36049] regionserver.Replication(203): jenkins-hbase16.apache.org,36049,1676290923148 started 2023-02-13 12:22:04,608 INFO [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(1638): Serving as jenkins-hbase16.apache.org,36049,1676290923148, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:36049, sessionid=0x10052d701450003 2023-02-13 12:22:04,608 INFO [RS:0;jenkins-hbase16:34139] regionserver.Replication(203): jenkins-hbase16.apache.org,34139,1676290923061 started 2023-02-13 12:22:04,608 INFO [RS:0;jenkins-hbase16:34139] regionserver.HRegionServer(1638): Serving as jenkins-hbase16.apache.org,34139,1676290923061, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:34139, sessionid=0x10052d701450001 2023-02-13 12:22:04,608 DEBUG [RS:0;jenkins-hbase16:34139] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 12:22:04,608 DEBUG [RS:2;jenkins-hbase16:36049] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 12:22:04,608 DEBUG [RS:0;jenkins-hbase16:34139] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:04,608 DEBUG [RS:2;jenkins-hbase16:36049] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:04,609 DEBUG [RS:0;jenkins-hbase16:34139] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,34139,1676290923061' 2023-02-13 12:22:04,609 DEBUG [RS:2;jenkins-hbase16:36049] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,36049,1676290923148' 2023-02-13 12:22:04,610 DEBUG [RS:0;jenkins-hbase16:34139] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 12:22:04,610 DEBUG [RS:2;jenkins-hbase16:36049] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 12:22:04,610 DEBUG [RS:0;jenkins-hbase16:34139] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 12:22:04,610 DEBUG [RS:2;jenkins-hbase16:36049] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 12:22:04,611 DEBUG [RS:2;jenkins-hbase16:36049] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 12:22:04,611 DEBUG [RS:0;jenkins-hbase16:34139] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 12:22:04,611 DEBUG [RS:0;jenkins-hbase16:34139] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 12:22:04,611 DEBUG [RS:0;jenkins-hbase16:34139] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:04,611 DEBUG [RS:0;jenkins-hbase16:34139] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,34139,1676290923061' 2023-02-13 12:22:04,611 DEBUG [RS:0;jenkins-hbase16:34139] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 12:22:04,611 DEBUG [RS:2;jenkins-hbase16:36049] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 12:22:04,611 DEBUG [RS:2;jenkins-hbase16:36049] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:04,616 INFO [RS:1;jenkins-hbase16:34079] regionserver.Replication(203): jenkins-hbase16.apache.org,34079,1676290923111 started 2023-02-13 12:22:04,616 INFO [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(1638): Serving as jenkins-hbase16.apache.org,34079,1676290923111, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:34079, sessionid=0x10052d701450002 2023-02-13 12:22:04,616 DEBUG [RS:2;jenkins-hbase16:36049] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,36049,1676290923148' 2023-02-13 12:22:04,616 DEBUG [RS:1;jenkins-hbase16:34079] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 12:22:04,616 DEBUG [RS:0;jenkins-hbase16:34139] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 12:22:04,616 DEBUG [RS:1;jenkins-hbase16:34079] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:04,616 DEBUG [RS:2;jenkins-hbase16:36049] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 12:22:04,616 DEBUG [RS:1;jenkins-hbase16:34079] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,34079,1676290923111' 2023-02-13 12:22:04,617 DEBUG [RS:1;jenkins-hbase16:34079] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 12:22:04,617 DEBUG [RS:0;jenkins-hbase16:34139] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 12:22:04,617 DEBUG [RS:2;jenkins-hbase16:36049] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 12:22:04,617 INFO [RS:0;jenkins-hbase16:34139] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 12:22:04,617 DEBUG [RS:1;jenkins-hbase16:34079] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 12:22:04,617 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 12:22:04,617 INFO [RS:0;jenkins-hbase16:34139] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 12:22:04,617 DEBUG [RS:2;jenkins-hbase16:36049] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 12:22:04,617 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 12:22:04,619 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 12:22:04,618 INFO [RS:2;jenkins-hbase16:36049] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 12:22:04,619 DEBUG [RS:1;jenkins-hbase16:34079] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 12:22:04,619 INFO [RS:2;jenkins-hbase16:36049] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 12:22:04,619 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 12:22:04,619 DEBUG [RS:1;jenkins-hbase16:34079] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 12:22:04,621 DEBUG [RS:1;jenkins-hbase16:34079] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:04,621 DEBUG [RS:1;jenkins-hbase16:34079] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,34079,1676290923111' 2023-02-13 12:22:04,621 DEBUG [RS:1;jenkins-hbase16:34079] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 12:22:04,622 DEBUG [RS:1;jenkins-hbase16:34079] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 12:22:04,622 DEBUG [RS:1;jenkins-hbase16:34079] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 12:22:04,622 INFO [RS:1;jenkins-hbase16:34079] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 12:22:04,622 INFO [RS:1;jenkins-hbase16:34079] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 12:22:04,622 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,34079,1676290923111, state=OPENING 2023-02-13 12:22:04,636 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-13 12:22:04,644 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 12:22:04,645 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 12:22:04,648 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:04,734 INFO [RS:2;jenkins-hbase16:36049] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C36049%2C1676290923148, suffix=, logDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/WALs/jenkins-hbase16.apache.org,36049,1676290923148, archiveDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/oldWALs, maxLogs=32 2023-02-13 12:22:04,735 INFO [RS:0;jenkins-hbase16:34139] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C34139%2C1676290923061, suffix=, logDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/WALs/jenkins-hbase16.apache.org,34139,1676290923061, archiveDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/oldWALs, maxLogs=32 2023-02-13 12:22:04,735 INFO [RS:1;jenkins-hbase16:34079] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C34079%2C1676290923111, suffix=, logDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/WALs/jenkins-hbase16.apache.org,34079,1676290923111, archiveDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/oldWALs, maxLogs=32 2023-02-13 12:22:04,761 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39681,DS-bbcaef41-1b8b-4853-a86f-103f917b438e,DISK] 2023-02-13 12:22:04,761 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36969,DS-618d2db0-bb39-4e38-8988-eb9c8891f87e,DISK] 2023-02-13 12:22:04,761 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39855,DS-c60bd1cd-ef15-4483-9b0e-1e3428eeb8f8,DISK] 2023-02-13 12:22:04,767 INFO [RS:1;jenkins-hbase16:34079] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/WALs/jenkins-hbase16.apache.org,34079,1676290923111/jenkins-hbase16.apache.org%2C34079%2C1676290923111.1676290924737 2023-02-13 12:22:04,771 DEBUG [RS:1;jenkins-hbase16:34079] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39681,DS-bbcaef41-1b8b-4853-a86f-103f917b438e,DISK], DatanodeInfoWithStorage[127.0.0.1:39855,DS-c60bd1cd-ef15-4483-9b0e-1e3428eeb8f8,DISK], DatanodeInfoWithStorage[127.0.0.1:36969,DS-618d2db0-bb39-4e38-8988-eb9c8891f87e,DISK]] 2023-02-13 12:22:04,772 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36969,DS-618d2db0-bb39-4e38-8988-eb9c8891f87e,DISK] 2023-02-13 12:22:04,772 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39681,DS-bbcaef41-1b8b-4853-a86f-103f917b438e,DISK] 2023-02-13 12:22:04,772 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39855,DS-c60bd1cd-ef15-4483-9b0e-1e3428eeb8f8,DISK] 2023-02-13 12:22:04,780 INFO [RS:0;jenkins-hbase16:34139] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/WALs/jenkins-hbase16.apache.org,34139,1676290923061/jenkins-hbase16.apache.org%2C34139%2C1676290923061.1676290924738 2023-02-13 12:22:04,785 DEBUG [RS:0;jenkins-hbase16:34139] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36969,DS-618d2db0-bb39-4e38-8988-eb9c8891f87e,DISK], DatanodeInfoWithStorage[127.0.0.1:39681,DS-bbcaef41-1b8b-4853-a86f-103f917b438e,DISK], DatanodeInfoWithStorage[127.0.0.1:39855,DS-c60bd1cd-ef15-4483-9b0e-1e3428eeb8f8,DISK]] 2023-02-13 12:22:04,786 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39855,DS-c60bd1cd-ef15-4483-9b0e-1e3428eeb8f8,DISK] 2023-02-13 12:22:04,787 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39681,DS-bbcaef41-1b8b-4853-a86f-103f917b438e,DISK] 2023-02-13 12:22:04,787 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36969,DS-618d2db0-bb39-4e38-8988-eb9c8891f87e,DISK] 2023-02-13 12:22:04,793 INFO [RS:2;jenkins-hbase16:36049] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/WALs/jenkins-hbase16.apache.org,36049,1676290923148/jenkins-hbase16.apache.org%2C36049%2C1676290923148.1676290924738 2023-02-13 12:22:04,793 DEBUG [RS:2;jenkins-hbase16:36049] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36969,DS-618d2db0-bb39-4e38-8988-eb9c8891f87e,DISK], DatanodeInfoWithStorage[127.0.0.1:39681,DS-bbcaef41-1b8b-4853-a86f-103f917b438e,DISK], DatanodeInfoWithStorage[127.0.0.1:39855,DS-c60bd1cd-ef15-4483-9b0e-1e3428eeb8f8,DISK]] 2023-02-13 12:22:04,829 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:04,831 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 12:22:04,834 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57844, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 12:22:04,847 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 12:22:04,848 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 12:22:04,852 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C34079%2C1676290923111.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/WALs/jenkins-hbase16.apache.org,34079,1676290923111, archiveDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/oldWALs, maxLogs=32 2023-02-13 12:22:04,872 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39681,DS-bbcaef41-1b8b-4853-a86f-103f917b438e,DISK] 2023-02-13 12:22:04,874 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36969,DS-618d2db0-bb39-4e38-8988-eb9c8891f87e,DISK] 2023-02-13 12:22:04,874 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39855,DS-c60bd1cd-ef15-4483-9b0e-1e3428eeb8f8,DISK] 2023-02-13 12:22:04,883 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/WALs/jenkins-hbase16.apache.org,34079,1676290923111/jenkins-hbase16.apache.org%2C34079%2C1676290923111.meta.1676290924854.meta 2023-02-13 12:22:04,883 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39681,DS-bbcaef41-1b8b-4853-a86f-103f917b438e,DISK], DatanodeInfoWithStorage[127.0.0.1:36969,DS-618d2db0-bb39-4e38-8988-eb9c8891f87e,DISK], DatanodeInfoWithStorage[127.0.0.1:39855,DS-c60bd1cd-ef15-4483-9b0e-1e3428eeb8f8,DISK]] 2023-02-13 12:22:04,883 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 12:22:04,885 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 12:22:04,903 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 12:22:04,909 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 12:22:04,914 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 12:22:04,914 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:04,914 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 12:22:04,914 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 12:22:04,917 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 12:22:04,919 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/info 2023-02-13 12:22:04,919 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/info 2023-02-13 12:22:04,919 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 12:22:04,920 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:04,920 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 12:22:04,922 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/rep_barrier 2023-02-13 12:22:04,922 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/rep_barrier 2023-02-13 12:22:04,922 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 12:22:04,923 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:04,923 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 12:22:04,925 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/table 2023-02-13 12:22:04,925 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/table 2023-02-13 12:22:04,925 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 12:22:04,926 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:04,929 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740 2023-02-13 12:22:04,932 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740 2023-02-13 12:22:04,935 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 12:22:04,938 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 12:22:04,939 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60456629, jitterRate=-0.09912602603435516}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 12:22:04,939 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676290924914Writing region info on filesystem at 1676290924914Initializing all the Stores at 1676290924916 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676290924916Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676290924916Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676290924917 (+1 ms)Cleaning up temporary data from old regions at 1676290924934 (+17 ms)Cleaning up detritus from prior splits at 1676290924934Running coprocessor post-open hooks at 1676290924939 (+5 ms)Region opened successfully at 1676290924939 2023-02-13 12:22:04,949 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676290924823 2023-02-13 12:22:04,965 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 12:22:04,966 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 12:22:04,966 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,34079,1676290923111, state=OPEN 2023-02-13 12:22:05,003 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 12:22:05,003 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 12:22:05,015 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-13 12:22:05,016 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,34079,1676290923111 in 355 msec 2023-02-13 12:22:05,022 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-13 12:22:05,022 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 591 msec 2023-02-13 12:22:05,027 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 842 msec 2023-02-13 12:22:05,027 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676290923209, completionTime=-1 2023-02-13 12:22:05,027 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 12:22:05,027 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 12:22:05,083 DEBUG [hconnection-0x8f0548d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:05,087 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57848, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:05,100 INFO [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 12:22:05,101 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676290985101 2023-02-13 12:22:05,101 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676291045101 2023-02-13 12:22:05,101 INFO [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 73 msec 2023-02-13 12:22:05,144 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,46677,1676290921946-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:05,144 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,46677,1676290921946-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:05,144 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,46677,1676290921946-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:05,146 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase16:46677, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:05,146 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 12:22:05,157 DEBUG [master/jenkins-hbase16:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 12:22:05,164 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-13 12:22:05,165 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 12:22:05,174 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-13 12:22:05,176 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 12:22:05,178 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 12:22:05,199 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/hbase/namespace/1c79ba080d1eda25c45efcc8579418da 2023-02-13 12:22:05,201 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/hbase/namespace/1c79ba080d1eda25c45efcc8579418da empty. 2023-02-13 12:22:05,202 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/hbase/namespace/1c79ba080d1eda25c45efcc8579418da 2023-02-13 12:22:05,202 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-13 12:22:05,241 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-13 12:22:05,244 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 1c79ba080d1eda25c45efcc8579418da, NAME => 'hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:05,261 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:05,261 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 1c79ba080d1eda25c45efcc8579418da, disabling compactions & flushes 2023-02-13 12:22:05,261 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. 2023-02-13 12:22:05,261 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. 2023-02-13 12:22:05,261 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. after waiting 0 ms 2023-02-13 12:22:05,261 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. 2023-02-13 12:22:05,261 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. 2023-02-13 12:22:05,261 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 1c79ba080d1eda25c45efcc8579418da: Waiting for close lock at 1676290925261Disabling compacts and flushes for region at 1676290925261Disabling writes for close at 1676290925261Writing region close event to WAL at 1676290925261Closed at 1676290925261 2023-02-13 12:22:05,266 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 12:22:05,282 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676290925269"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290925269"}]},"ts":"1676290925269"} 2023-02-13 12:22:05,304 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 12:22:05,306 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 12:22:05,311 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290925307"}]},"ts":"1676290925307"} 2023-02-13 12:22:05,316 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-13 12:22:05,337 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 12:22:05,338 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 12:22:05,338 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 12:22:05,338 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 12:22:05,339 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 12:22:05,342 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1c79ba080d1eda25c45efcc8579418da, ASSIGN}] 2023-02-13 12:22:05,349 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1c79ba080d1eda25c45efcc8579418da, ASSIGN 2023-02-13 12:22:05,351 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=1c79ba080d1eda25c45efcc8579418da, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=false 2023-02-13 12:22:05,504 INFO [jenkins-hbase16:46677] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 12:22:05,505 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=1c79ba080d1eda25c45efcc8579418da, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:05,506 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676290925505"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290925505"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290925505"}]},"ts":"1676290925505"} 2023-02-13 12:22:05,511 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 1c79ba080d1eda25c45efcc8579418da, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:05,665 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:05,666 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 12:22:05,671 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:44412, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 12:22:05,679 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. 2023-02-13 12:22:05,681 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1c79ba080d1eda25c45efcc8579418da, NAME => 'hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da.', STARTKEY => '', ENDKEY => ''} 2023-02-13 12:22:05,681 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 1c79ba080d1eda25c45efcc8579418da 2023-02-13 12:22:05,681 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:05,682 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 1c79ba080d1eda25c45efcc8579418da 2023-02-13 12:22:05,682 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 1c79ba080d1eda25c45efcc8579418da 2023-02-13 12:22:05,684 INFO [StoreOpener-1c79ba080d1eda25c45efcc8579418da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1c79ba080d1eda25c45efcc8579418da 2023-02-13 12:22:05,686 DEBUG [StoreOpener-1c79ba080d1eda25c45efcc8579418da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/namespace/1c79ba080d1eda25c45efcc8579418da/info 2023-02-13 12:22:05,686 DEBUG [StoreOpener-1c79ba080d1eda25c45efcc8579418da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/namespace/1c79ba080d1eda25c45efcc8579418da/info 2023-02-13 12:22:05,686 INFO [StoreOpener-1c79ba080d1eda25c45efcc8579418da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1c79ba080d1eda25c45efcc8579418da columnFamilyName info 2023-02-13 12:22:05,687 INFO [StoreOpener-1c79ba080d1eda25c45efcc8579418da-1] regionserver.HStore(310): Store=1c79ba080d1eda25c45efcc8579418da/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:05,689 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/namespace/1c79ba080d1eda25c45efcc8579418da 2023-02-13 12:22:05,690 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/namespace/1c79ba080d1eda25c45efcc8579418da 2023-02-13 12:22:05,694 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 1c79ba080d1eda25c45efcc8579418da 2023-02-13 12:22:05,699 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/namespace/1c79ba080d1eda25c45efcc8579418da/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:05,700 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 1c79ba080d1eda25c45efcc8579418da; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73622149, jitterRate=0.09705550968647003}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:05,700 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 1c79ba080d1eda25c45efcc8579418da: Running coprocessor pre-open hook at 1676290925682Writing region info on filesystem at 1676290925682Initializing all the Stores at 1676290925683 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676290925683Cleaning up temporary data from old regions at 1676290925691 (+8 ms)Cleaning up detritus from prior splits at 1676290925691Running coprocessor post-open hooks at 1676290925700 (+9 ms)Region opened successfully at 1676290925700 2023-02-13 12:22:05,702 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da., pid=6, masterSystemTime=1676290925665 2023-02-13 12:22:05,706 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. 2023-02-13 12:22:05,707 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. 2023-02-13 12:22:05,708 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=1c79ba080d1eda25c45efcc8579418da, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:05,709 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676290925708"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290925708"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290925708"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290925708"}]},"ts":"1676290925708"} 2023-02-13 12:22:05,716 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-13 12:22:05,716 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 1c79ba080d1eda25c45efcc8579418da, server=jenkins-hbase16.apache.org,36049,1676290923148 in 201 msec 2023-02-13 12:22:05,719 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-13 12:22:05,720 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=1c79ba080d1eda25c45efcc8579418da, ASSIGN in 374 msec 2023-02-13 12:22:05,721 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 12:22:05,721 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290925721"}]},"ts":"1676290925721"} 2023-02-13 12:22:05,724 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-13 12:22:05,746 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 12:22:05,749 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 580 msec 2023-02-13 12:22:05,778 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-13 12:22:05,786 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-13 12:22:05,786 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 12:22:05,813 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:05,814 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:44422, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:05,826 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-13 12:22:05,849 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 12:22:05,875 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 54 msec 2023-02-13 12:22:05,881 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-13 12:22:05,903 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 12:22:05,915 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 34 msec 2023-02-13 12:22:05,941 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 12:22:05,966 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 12:22:05,969 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.735sec 2023-02-13 12:22:05,974 INFO [master/jenkins-hbase16:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 12:22:05,977 INFO [master/jenkins-hbase16:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 12:22:05,977 INFO [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 12:22:05,979 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,46677,1676290921946-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 12:22:05,981 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,46677,1676290921946-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 12:22:05,985 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ReadOnlyZKClient(139): Connect 0x20c4cf78 to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:05,990 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 12:22:05,995 DEBUG [Listener at localhost.localdomain/46179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a042592, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:06,011 DEBUG [hconnection-0x39338c12-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:06,025 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57854, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:06,034 INFO [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase16.apache.org,46677,1676290921946 2023-02-13 12:22:06,063 INFO [Listener at localhost.localdomain/46179] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=361, OpenFileDescriptor=692, MaxFileDescriptor=60000, SystemLoadAverage=245, ProcessCount=171, AvailableMemoryMB=5026 2023-02-13 12:22:06,077 DEBUG [Listener at localhost.localdomain/46179] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 12:22:06,081 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39376, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 12:22:06,093 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$4(2102): Client=jenkins//188.40.62.62 create 'testRestoreSnapshot-1676290926065', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-13 12:22:06,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:06,098 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676290926065 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 12:22:06,101 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676290926065 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 12:22:06,103 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676290926065" procId is: 9 2023-02-13 12:22:06,110 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:06,110 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:06,110 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:06,110 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:06,110 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:06,110 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:06,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 12:22:06,111 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 empty. 2023-02-13 12:22:06,111 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 empty. 2023-02-13 12:22:06,111 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 empty. 2023-02-13 12:22:06,111 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 empty. 2023-02-13 12:22:06,111 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c empty. 2023-02-13 12:22:06,113 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:06,113 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:06,114 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:06,114 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:06,118 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 empty. 2023-02-13 12:22:06,118 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:06,119 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:06,119 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676290926065 regions 2023-02-13 12:22:06,141 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/.tabledesc/.tableinfo.0000000001 2023-02-13 12:22:06,143 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(7671): creating {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676290926065', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:06,144 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(7671): creating {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676290926065', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:06,144 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(7671): creating {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676290926065', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:06,221 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:06,221 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1603): Closing 50cc7d0236be777dd672124c8ff93b56, disabling compactions & flushes 2023-02-13 12:22:06,222 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:06,222 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:06,222 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:06,225 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1603): Closing 6d6fd747f58897a134fa0a460eb9ec99, disabling compactions & flushes 2023-02-13 12:22:06,225 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. after waiting 0 ms 2023-02-13 12:22:06,225 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:06,225 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:06,225 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. after waiting 0 ms 2023-02-13 12:22:06,225 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:06,225 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:06,225 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:06,225 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:06,225 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1557): Region close journal for 6d6fd747f58897a134fa0a460eb9ec99: Waiting for close lock at 1676290926225Disabling compacts and flushes for region at 1676290926225Disabling writes for close at 1676290926225Writing region close event to WAL at 1676290926225Closed at 1676290926225 2023-02-13 12:22:06,225 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1557): Region close journal for 50cc7d0236be777dd672124c8ff93b56: Waiting for close lock at 1676290926221Disabling compacts and flushes for region at 1676290926221Disabling writes for close at 1676290926225 (+4 ms)Writing region close event to WAL at 1676290926225Closed at 1676290926225 2023-02-13 12:22:06,227 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(7671): creating {ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676290926065', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:06,227 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(7671): creating {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676290926065', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:06,230 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:06,231 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1603): Closing 6e9099611b20e9378ce559b4b3a17188, disabling compactions & flushes 2023-02-13 12:22:06,231 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:06,231 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:06,231 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. after waiting 0 ms 2023-02-13 12:22:06,231 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:06,231 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:06,231 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1557): Region close journal for 6e9099611b20e9378ce559b4b3a17188: Waiting for close lock at 1676290926231Disabling compacts and flushes for region at 1676290926231Disabling writes for close at 1676290926231Writing region close event to WAL at 1676290926231Closed at 1676290926231 2023-02-13 12:22:06,232 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(7671): creating {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676290926065', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:06,257 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:06,257 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1603): Closing 197a58e90a6010ab47efab725a975e4c, disabling compactions & flushes 2023-02-13 12:22:06,257 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:06,258 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:06,258 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. after waiting 0 ms 2023-02-13 12:22:06,258 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:06,258 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:06,258 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-1] regionserver.HRegion(1557): Region close journal for 197a58e90a6010ab47efab725a975e4c: Waiting for close lock at 1676290926257Disabling compacts and flushes for region at 1676290926257Disabling writes for close at 1676290926258 (+1 ms)Writing region close event to WAL at 1676290926258Closed at 1676290926258 2023-02-13 12:22:06,259 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:06,260 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1603): Closing 81a0069ff66a08b05e0e5fe4210e0ce2, disabling compactions & flushes 2023-02-13 12:22:06,260 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:06,260 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:06,260 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. after waiting 0 ms 2023-02-13 12:22:06,260 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:06,260 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:06,260 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-2] regionserver.HRegion(1557): Region close journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Waiting for close lock at 1676290926260Disabling compacts and flushes for region at 1676290926260Disabling writes for close at 1676290926260Writing region close event to WAL at 1676290926260Closed at 1676290926260 2023-02-13 12:22:06,265 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:06,265 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1603): Closing 585751c2808ad67a24a61f19e9ac17f5, disabling compactions & flushes 2023-02-13 12:22:06,265 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:06,265 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:06,265 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. after waiting 0 ms 2023-02-13 12:22:06,265 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:06,265 INFO [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:06,265 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676290926065-pool-0] regionserver.HRegion(1557): Region close journal for 585751c2808ad67a24a61f19e9ac17f5: Waiting for close lock at 1676290926265Disabling compacts and flushes for region at 1676290926265Disabling writes for close at 1676290926265Writing region close event to WAL at 1676290926265Closed at 1676290926265 2023-02-13 12:22:06,271 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676290926065 execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 12:22:06,272 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290926272"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290926272"}]},"ts":"1676290926272"} 2023-02-13 12:22:06,272 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290926272"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290926272"}]},"ts":"1676290926272"} 2023-02-13 12:22:06,273 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290926272"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290926272"}]},"ts":"1676290926272"} 2023-02-13 12:22:06,273 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290926272"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290926272"}]},"ts":"1676290926272"} 2023-02-13 12:22:06,273 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290926272"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290926272"}]},"ts":"1676290926272"} 2023-02-13 12:22:06,273 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290926272"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290926272"}]},"ts":"1676290926272"} 2023-02-13 12:22:06,317 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 12:22:06,319 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676290926065 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 12:22:06,320 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290926320"}]},"ts":"1676290926320"} 2023-02-13 12:22:06,322 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLING in hbase:meta 2023-02-13 12:22:06,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 12:22:06,379 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 12:22:06,387 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 12:22:06,387 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 12:22:06,387 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 12:22:06,387 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 12:22:06,388 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN}] 2023-02-13 12:22:06,392 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN 2023-02-13 12:22:06,392 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN 2023-02-13 12:22:06,393 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN 2023-02-13 12:22:06,393 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN 2023-02-13 12:22:06,395 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=false 2023-02-13 12:22:06,395 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=false 2023-02-13 12:22:06,395 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN 2023-02-13 12:22:06,395 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=false 2023-02-13 12:22:06,395 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=false 2023-02-13 12:22:06,396 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=false 2023-02-13 12:22:06,396 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN 2023-02-13 12:22:06,398 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=false 2023-02-13 12:22:06,545 INFO [jenkins-hbase16:46677] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 12:22:06,553 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:06,553 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:06,553 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:06,553 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:06,554 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290926553"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290926553"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290926553"}]},"ts":"1676290926553"} 2023-02-13 12:22:06,554 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:06,555 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290926553"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290926553"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290926553"}]},"ts":"1676290926553"} 2023-02-13 12:22:06,554 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290926553"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290926553"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290926553"}]},"ts":"1676290926553"} 2023-02-13 12:22:06,554 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290926553"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290926553"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290926553"}]},"ts":"1676290926553"} 2023-02-13 12:22:06,555 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290926553"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290926553"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290926553"}]},"ts":"1676290926553"} 2023-02-13 12:22:06,559 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=13, state=RUNNABLE; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:06,561 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=10, state=RUNNABLE; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:06,564 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:06,564 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290926564"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290926564"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290926564"}]},"ts":"1676290926564"} 2023-02-13 12:22:06,565 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=15, state=RUNNABLE; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:06,567 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:06,568 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=11, state=RUNNABLE; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:06,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=14, state=RUNNABLE; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:06,712 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:06,712 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 12:22:06,715 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:48058, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 12:22:06,721 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:06,721 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:06,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:22:06,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:22:06,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:06,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:06,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:06,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:06,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:06,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:06,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:06,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:06,726 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:06,732 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:06,732 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:06,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:22:06,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:06,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:06,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:06,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:06,734 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:06,734 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:06,735 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:06,735 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:06,735 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:06,737 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 50cc7d0236be777dd672124c8ff93b56 columnFamilyName cf 2023-02-13 12:22:06,737 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6e9099611b20e9378ce559b4b3a17188 columnFamilyName cf 2023-02-13 12:22:06,737 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:06,738 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:06,738 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d6fd747f58897a134fa0a460eb9ec99 columnFamilyName cf 2023-02-13 12:22:06,740 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(310): Store=50cc7d0236be777dd672124c8ff93b56/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:06,740 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(310): Store=6d6fd747f58897a134fa0a460eb9ec99/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:06,740 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(310): Store=6e9099611b20e9378ce559b4b3a17188/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:06,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:06,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:06,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:06,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:06,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:06,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:06,749 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:06,749 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:06,750 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:06,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:06,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:06,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:06,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 50cc7d0236be777dd672124c8ff93b56; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64112431, jitterRate=-0.044650331139564514}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:06,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 50cc7d0236be777dd672124c8ff93b56: Running coprocessor pre-open hook at 1676290926723Writing region info on filesystem at 1676290926723Initializing all the Stores at 1676290926724 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290926724Cleaning up temporary data from old regions at 1676290926746 (+22 ms)Cleaning up detritus from prior splits at 1676290926746Running coprocessor post-open hooks at 1676290926759 (+13 ms)Region opened successfully at 1676290926759 2023-02-13 12:22:06,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6d6fd747f58897a134fa0a460eb9ec99; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59224162, jitterRate=-0.117491215467453}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:06,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6e9099611b20e9378ce559b4b3a17188; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69429911, jitterRate=0.03458629548549652}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:06,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6d6fd747f58897a134fa0a460eb9ec99: Running coprocessor pre-open hook at 1676290926734Writing region info on filesystem at 1676290926734Initializing all the Stores at 1676290926735 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290926735Cleaning up temporary data from old regions at 1676290926746 (+11 ms)Cleaning up detritus from prior splits at 1676290926746Running coprocessor post-open hooks at 1676290926760 (+14 ms)Region opened successfully at 1676290926760 2023-02-13 12:22:06,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6e9099611b20e9378ce559b4b3a17188: Running coprocessor pre-open hook at 1676290926723Writing region info on filesystem at 1676290926723Initializing all the Stores at 1676290926724 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290926725 (+1 ms)Cleaning up temporary data from old regions at 1676290926746 (+21 ms)Cleaning up detritus from prior splits at 1676290926746Running coprocessor post-open hooks at 1676290926760 (+14 ms)Region opened successfully at 1676290926760 2023-02-13 12:22:06,762 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56., pid=20, masterSystemTime=1676290926712 2023-02-13 12:22:06,766 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99., pid=19, masterSystemTime=1676290926721 2023-02-13 12:22:06,766 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188., pid=17, masterSystemTime=1676290926714 2023-02-13 12:22:06,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:06,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:06,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:06,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:22:06,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:06,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:06,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:06,772 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:06,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:06,772 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290926772"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290926772"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290926772"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290926772"}]},"ts":"1676290926772"} 2023-02-13 12:22:06,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:06,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:06,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:06,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:22:06,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:06,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:06,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:06,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:06,775 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:06,776 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290926775"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290926775"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290926775"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290926775"}]},"ts":"1676290926775"} 2023-02-13 12:22:06,775 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:06,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:06,778 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:06,780 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:06,780 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:06,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:22:06,781 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:06,781 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290926780"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290926780"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290926780"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290926780"}]},"ts":"1676290926780"} 2023-02-13 12:22:06,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:06,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:06,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:06,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:06,781 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:06,782 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:06,783 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 197a58e90a6010ab47efab725a975e4c columnFamilyName cf 2023-02-13 12:22:06,785 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:06,785 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:06,786 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(310): Store=197a58e90a6010ab47efab725a975e4c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:06,786 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 585751c2808ad67a24a61f19e9ac17f5 columnFamilyName cf 2023-02-13 12:22:06,789 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:06,786 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-02-13 12:22:06,792 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111 in 210 msec 2023-02-13 12:22:06,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:06,795 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(310): Store=585751c2808ad67a24a61f19e9ac17f5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:06,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:06,796 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN in 398 msec 2023-02-13 12:22:06,797 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=10 2023-02-13 12:22:06,797 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=10, state=SUCCESS; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148 in 220 msec 2023-02-13 12:22:06,798 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=11 2023-02-13 12:22:06,798 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=11, state=SUCCESS; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061 in 218 msec 2023-02-13 12:22:06,800 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN in 409 msec 2023-02-13 12:22:06,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:06,802 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN in 410 msec 2023-02-13 12:22:06,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:06,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:06,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:06,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:06,807 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:06,807 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:06,808 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 197a58e90a6010ab47efab725a975e4c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65355412, jitterRate=-0.02612847089767456}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:06,808 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81a0069ff66a08b05e0e5fe4210e0ce2 columnFamilyName cf 2023-02-13 12:22:06,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 197a58e90a6010ab47efab725a975e4c: Running coprocessor pre-open hook at 1676290926772Writing region info on filesystem at 1676290926772Initializing all the Stores at 1676290926774 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290926774Cleaning up temporary data from old regions at 1676290926798 (+24 ms)Cleaning up detritus from prior splits at 1676290926798Running coprocessor post-open hooks at 1676290926808 (+10 ms)Region opened successfully at 1676290926808 2023-02-13 12:22:06,810 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c., pid=21, masterSystemTime=1676290926721 2023-02-13 12:22:06,810 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(310): Store=81a0069ff66a08b05e0e5fe4210e0ce2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:06,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:06,813 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 585751c2808ad67a24a61f19e9ac17f5; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63652386, jitterRate=-0.05150553584098816}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:06,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 585751c2808ad67a24a61f19e9ac17f5: Running coprocessor pre-open hook at 1676290926775Writing region info on filesystem at 1676290926775Initializing all the Stores at 1676290926777 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290926777Cleaning up temporary data from old regions at 1676290926803 (+26 ms)Cleaning up detritus from prior splits at 1676290926804 (+1 ms)Running coprocessor post-open hooks at 1676290926813 (+9 ms)Region opened successfully at 1676290926813 2023-02-13 12:22:06,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:06,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:06,814 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5., pid=18, masterSystemTime=1676290926714 2023-02-13 12:22:06,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:06,815 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:06,819 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:06,819 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290926819"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290926819"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290926819"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290926819"}]},"ts":"1676290926819"} 2023-02-13 12:22:06,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:06,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:06,824 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:06,825 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:06,825 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290926825"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290926825"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290926825"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290926825"}]},"ts":"1676290926825"} 2023-02-13 12:22:06,827 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=14 2023-02-13 12:22:06,828 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=14, state=SUCCESS; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111 in 253 msec 2023-02-13 12:22:06,830 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN in 440 msec 2023-02-13 12:22:06,834 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN in 443 msec 2023-02-13 12:22:06,831 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=15 2023-02-13 12:22:06,837 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=15, state=SUCCESS; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148 in 263 msec 2023-02-13 12:22:06,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:06,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81a0069ff66a08b05e0e5fe4210e0ce2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72013863, jitterRate=0.07309018075466156}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:06,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Running coprocessor pre-open hook at 1676290926781Writing region info on filesystem at 1676290926781Initializing all the Stores at 1676290926785 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290926785Cleaning up temporary data from old regions at 1676290926816 (+31 ms)Cleaning up detritus from prior splits at 1676290926818 (+2 ms)Running coprocessor post-open hooks at 1676290926838 (+20 ms)Region opened successfully at 1676290926838 2023-02-13 12:22:06,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2., pid=16, masterSystemTime=1676290926712 2023-02-13 12:22:06,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:06,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:06,843 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:06,843 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290926843"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290926843"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290926843"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290926843"}]},"ts":"1676290926843"} 2023-02-13 12:22:06,849 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=13 2023-02-13 12:22:06,849 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=13, state=SUCCESS; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061 in 287 msec 2023-02-13 12:22:06,856 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=9 2023-02-13 12:22:06,857 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN in 461 msec 2023-02-13 12:22:06,858 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676290926065 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 12:22:06,858 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290926858"}]},"ts":"1676290926858"} 2023-02-13 12:22:06,860 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLED in hbase:meta 2023-02-13 12:22:06,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 12:22:06,875 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676290926065 execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 12:22:06,879 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676290926065 in 783 msec 2023-02-13 12:22:07,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 12:22:07,622 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 9 completed 2023-02-13 12:22:07,628 DEBUG [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676290926065 2023-02-13 12:22:07,629 DEBUG [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:07,632 INFO [Listener at localhost.localdomain/46179] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 12:22:07,653 DEBUG [hconnection-0x39338c12-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:07,656 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:44426, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:07,660 DEBUG [hconnection-0x39338c12-shared-pool-5] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:07,664 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:48064, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:07,676 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676290926065 2023-02-13 12:22:07,680 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676290926065 2023-02-13 12:22:07,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:07,693 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290927693"}]},"ts":"1676290927693"} 2023-02-13 12:22:07,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-13 12:22:07,695 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLING in hbase:meta 2023-02-13 12:22:07,728 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676290926065 to state=DISABLING 2023-02-13 12:22:07,731 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN}] 2023-02-13 12:22:07,734 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN 2023-02-13 12:22:07,734 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN 2023-02-13 12:22:07,734 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN 2023-02-13 12:22:07,734 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN 2023-02-13 12:22:07,734 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN 2023-02-13 12:22:07,735 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:07,735 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:07,735 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:07,735 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:07,736 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290927735"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290927735"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290927735"}]},"ts":"1676290927735"} 2023-02-13 12:22:07,736 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290927735"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290927735"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290927735"}]},"ts":"1676290927735"} 2023-02-13 12:22:07,736 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290927735"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290927735"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290927735"}]},"ts":"1676290927735"} 2023-02-13 12:22:07,736 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:07,736 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290927735"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290927735"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290927735"}]},"ts":"1676290927735"} 2023-02-13 12:22:07,736 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290927736"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290927736"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290927736"}]},"ts":"1676290927736"} 2023-02-13 12:22:07,739 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=25, state=RUNNABLE; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:07,747 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=24, state=RUNNABLE; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:07,749 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=26, state=RUNNABLE; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:07,751 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=23, state=RUNNABLE; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:07,753 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=28, state=RUNNABLE; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:07,761 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN 2023-02-13 12:22:07,764 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:07,764 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290927764"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290927764"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290927764"}]},"ts":"1676290927764"} 2023-02-13 12:22:07,767 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=27, state=RUNNABLE; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:07,908 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:07,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 197a58e90a6010ab47efab725a975e4c, disabling compactions & flushes 2023-02-13 12:22:07,909 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:07,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:07,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. after waiting 0 ms 2023-02-13 12:22:07,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:07,911 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:07,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 585751c2808ad67a24a61f19e9ac17f5, disabling compactions & flushes 2023-02-13 12:22:07,912 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:07,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:07,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. after waiting 0 ms 2023-02-13 12:22:07,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:07,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:07,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 50cc7d0236be777dd672124c8ff93b56, disabling compactions & flushes 2023-02-13 12:22:07,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:07,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:07,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. after waiting 0 ms 2023-02-13 12:22:07,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:07,925 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:22:07,926 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:22:07,927 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:07,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 197a58e90a6010ab47efab725a975e4c: Waiting for close lock at 1676290927909Running coprocessor pre-close hooks at 1676290927909Disabling compacts and flushes for region at 1676290927909Disabling writes for close at 1676290927909Writing region close event to WAL at 1676290927910 (+1 ms)Running coprocessor post-close hooks at 1676290927927 (+17 ms)Closed at 1676290927927 2023-02-13 12:22:07,929 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:07,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 585751c2808ad67a24a61f19e9ac17f5: Waiting for close lock at 1676290927911Running coprocessor pre-close hooks at 1676290927911Disabling compacts and flushes for region at 1676290927911Disabling writes for close at 1676290927912 (+1 ms)Writing region close event to WAL at 1676290927915 (+3 ms)Running coprocessor post-close hooks at 1676290927929 (+14 ms)Closed at 1676290927929 2023-02-13 12:22:07,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:07,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:07,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6d6fd747f58897a134fa0a460eb9ec99, disabling compactions & flushes 2023-02-13 12:22:07,931 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:07,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:07,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. after waiting 0 ms 2023-02-13 12:22:07,932 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:07,933 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSED 2023-02-13 12:22:07,934 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290927933"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290927933"}]},"ts":"1676290927933"} 2023-02-13 12:22:07,936 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:07,936 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:07,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6e9099611b20e9378ce559b4b3a17188, disabling compactions & flushes 2023-02-13 12:22:07,936 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:07,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:22:07,936 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSED 2023-02-13 12:22:07,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:07,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. after waiting 0 ms 2023-02-13 12:22:07,938 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290927936"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290927936"}]},"ts":"1676290927936"} 2023-02-13 12:22:07,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:07,943 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:07,943 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 50cc7d0236be777dd672124c8ff93b56: Waiting for close lock at 1676290927915Running coprocessor pre-close hooks at 1676290927915Disabling compacts and flushes for region at 1676290927915Disabling writes for close at 1676290927918 (+3 ms)Writing region close event to WAL at 1676290927919 (+1 ms)Running coprocessor post-close hooks at 1676290927942 (+23 ms)Closed at 1676290927943 (+1 ms) 2023-02-13 12:22:07,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-13 12:22:07,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:07,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:07,946 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=27 2023-02-13 12:22:07,946 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSED 2023-02-13 12:22:07,946 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=27, state=SUCCESS; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111 in 171 msec 2023-02-13 12:22:07,947 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290927946"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290927946"}]},"ts":"1676290927946"} 2023-02-13 12:22:07,949 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=28 2023-02-13 12:22:07,950 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=28, state=SUCCESS; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148 in 192 msec 2023-02-13 12:22:07,950 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN in 216 msec 2023-02-13 12:22:07,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81a0069ff66a08b05e0e5fe4210e0ce2, disabling compactions & flushes 2023-02-13 12:22:07,953 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:07,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:07,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. after waiting 0 ms 2023-02-13 12:22:07,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:07,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:22:07,955 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN in 218 msec 2023-02-13 12:22:07,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:07,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6d6fd747f58897a134fa0a460eb9ec99: Waiting for close lock at 1676290927930Running coprocessor pre-close hooks at 1676290927930Disabling compacts and flushes for region at 1676290927930Disabling writes for close at 1676290927932 (+2 ms)Writing region close event to WAL at 1676290927932Running coprocessor post-close hooks at 1676290927957 (+25 ms)Closed at 1676290927957 2023-02-13 12:22:07,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:22:07,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=24 2023-02-13 12:22:07,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=24, state=SUCCESS; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061 in 202 msec 2023-02-13 12:22:07,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:07,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6e9099611b20e9378ce559b4b3a17188: Waiting for close lock at 1676290927936Running coprocessor pre-close hooks at 1676290927936Disabling compacts and flushes for region at 1676290927936Disabling writes for close at 1676290927938 (+2 ms)Writing region close event to WAL at 1676290927938Running coprocessor post-close hooks at 1676290927959 (+21 ms)Closed at 1676290927959 2023-02-13 12:22:07,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:22:07,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:07,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:07,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Waiting for close lock at 1676290927946Running coprocessor pre-close hooks at 1676290927946Disabling compacts and flushes for region at 1676290927946Disabling writes for close at 1676290927953 (+7 ms)Writing region close event to WAL at 1676290927955 (+2 ms)Running coprocessor post-close hooks at 1676290927966 (+11 ms)Closed at 1676290927966 2023-02-13 12:22:07,968 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSED 2023-02-13 12:22:07,968 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN in 228 msec 2023-02-13 12:22:07,968 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290927968"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290927968"}]},"ts":"1676290927968"} 2023-02-13 12:22:07,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:07,969 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSED 2023-02-13 12:22:07,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:07,969 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290927969"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290927969"}]},"ts":"1676290927969"} 2023-02-13 12:22:07,970 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSED 2023-02-13 12:22:07,971 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290927970"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290927970"}]},"ts":"1676290927970"} 2023-02-13 12:22:07,974 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=25 2023-02-13 12:22:07,974 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=25, state=SUCCESS; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111 in 232 msec 2023-02-13 12:22:07,975 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=23 2023-02-13 12:22:07,975 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=23, state=SUCCESS; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148 in 221 msec 2023-02-13 12:22:07,977 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN in 243 msec 2023-02-13 12:22:07,977 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=26 2023-02-13 12:22:07,977 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN in 245 msec 2023-02-13 12:22:07,977 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=26, state=SUCCESS; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061 in 224 msec 2023-02-13 12:22:07,981 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=22 2023-02-13 12:22:07,981 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN in 246 msec 2023-02-13 12:22:07,983 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290927983"}]},"ts":"1676290927983"} 2023-02-13 12:22:07,985 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLED in hbase:meta 2023-02-13 12:22:08,032 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676290926065 to state=DISABLED 2023-02-13 12:22:08,038 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676290926065 in 352 msec 2023-02-13 12:22:08,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-13 12:22:08,448 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 22 completed 2023-02-13 12:22:08,462 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=emptySnaptb-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:08,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676290928462 (current time:1676290928462). 2023-02-13 12:22:08,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 12:22:08,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(139): Connect 0x5ee52be0 to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:08,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1959555a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:08,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:08,511 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57866, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:08,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ee52be0 to 127.0.0.1:62541 2023-02-13 12:22:08,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:08,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 12:22:08,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:08,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:08,531 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:08,532 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE; timeout=600000, timestamp=1676291528532 2023-02-13 12:22:08,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:08,532 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676290926065 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676290926065 2023-02-13 12:22:08,537 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 12:22:08,538 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 12:22:08,552 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } 2023-02-13 12:22:08,560 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:08,560 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:08,561 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:08,561 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:08,561 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:08,561 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:08,562 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:08,565 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:08,565 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:08,565 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:08,565 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:08,565 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:08,565 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:08,565 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:08,572 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:08,573 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:08,573 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:08,573 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:08,573 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:08,573 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:08,573 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:08,788 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:22:08,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } is done 2023-02-13 12:22:08,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 12:22:08,792 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 12:22:09,083 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } as finished. 2023-02-13 12:22:09,084 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676290926065 2023-02-13 12:22:09,086 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/emptySnaptb-1676290926065 2023-02-13 12:22:09,131 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/emptySnaptb-1676290926065 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/emptySnaptb-1676290926065 2023-02-13 12:22:09,162 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676290926065 of table testRestoreSnapshot-1676290926065 completed 2023-02-13 12:22:09,163 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/emptySnaptb-1676290926065 2023-02-13 12:22:09,163 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/emptySnaptb-1676290926065 2023-02-13 12:22:09,163 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676290926065 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676290926065 at 1676290928532Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } at 1676290928552 (+20 ms)Consolidate snapshot: emptySnaptb-1676290926065 at 1676290929084 (+532 ms)Loading Region manifests for emptySnaptb-1676290926065 at 1676290929084Writing data manifest for emptySnaptb-1676290926065 at 1676290929104 (+20 ms)Verifying snapshot: emptySnaptb-1676290926065 at 1676290929118 (+14 ms)Snapshot emptySnaptb-1676290926065 of table testRestoreSnapshot-1676290926065 completed at 1676290929162 (+44 ms) 2023-02-13 12:22:09,164 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:09,166 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE in 640 msec 2023-02-13 12:22:09,292 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:22:09,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } is done 2023-02-13 12:22:09,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 12:22:09,297 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676290926065 2023-02-13 12:22:09,302 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testRestoreSnapshot-1676290926065 2023-02-13 12:22:09,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:09,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 12:22:09,314 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676290926065 2023-02-13 12:22:09,314 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290929314"}]},"ts":"1676290929314"} 2023-02-13 12:22:09,316 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLING in hbase:meta 2023-02-13 12:22:09,329 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 12:22:09,331 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN}] 2023-02-13 12:22:09,334 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN 2023-02-13 12:22:09,334 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN 2023-02-13 12:22:09,334 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN 2023-02-13 12:22:09,334 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN 2023-02-13 12:22:09,335 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:09,335 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN 2023-02-13 12:22:09,335 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:09,335 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:09,335 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:09,336 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=true 2023-02-13 12:22:09,335 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=true 2023-02-13 12:22:09,335 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=true 2023-02-13 12:22:09,336 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=true 2023-02-13 12:22:09,337 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:09,337 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN 2023-02-13 12:22:09,337 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=true 2023-02-13 12:22:09,337 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:09,338 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=true 2023-02-13 12:22:09,486 INFO [jenkins-hbase16:46677] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 12:22:09,486 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:09,486 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:09,486 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:09,486 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290929486"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290929486"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290929486"}]},"ts":"1676290929486"} 2023-02-13 12:22:09,486 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:09,487 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290929486"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290929486"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290929486"}]},"ts":"1676290929486"} 2023-02-13 12:22:09,487 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290929486"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290929486"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290929486"}]},"ts":"1676290929486"} 2023-02-13 12:22:09,486 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290929486"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290929486"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290929486"}]},"ts":"1676290929486"} 2023-02-13 12:22:09,486 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:09,487 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290929486"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290929486"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290929486"}]},"ts":"1676290929486"} 2023-02-13 12:22:09,489 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:09,491 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=40, state=RUNNABLE; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:09,494 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=37, state=RUNNABLE; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:09,497 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=39, state=RUNNABLE; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:09,499 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=38, state=RUNNABLE; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:09,501 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:09,501 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290929501"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290929501"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290929501"}]},"ts":"1676290929501"} 2023-02-13 12:22:09,504 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=41, state=RUNNABLE; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:09,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 12:22:09,647 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:09,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:22:09,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:09,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:09,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:09,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:09,652 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:09,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:22:09,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:09,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:09,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:09,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:09,654 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:09,655 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:09,655 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:09,655 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:09,656 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 585751c2808ad67a24a61f19e9ac17f5 columnFamilyName cf 2023-02-13 12:22:09,657 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:09,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:09,657 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:09,658 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(310): Store=585751c2808ad67a24a61f19e9ac17f5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:09,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:22:09,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:09,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:09,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:09,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:09,660 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 50cc7d0236be777dd672124c8ff93b56 columnFamilyName cf 2023-02-13 12:22:09,661 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:09,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:09,665 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(310): Store=50cc7d0236be777dd672124c8ff93b56/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:09,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:09,668 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:09,668 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:09,668 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d6fd747f58897a134fa0a460eb9ec99 columnFamilyName cf 2023-02-13 12:22:09,669 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:09,670 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(310): Store=6d6fd747f58897a134fa0a460eb9ec99/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:09,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:09,673 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:09,673 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:09,673 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:09,675 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 585751c2808ad67a24a61f19e9ac17f5; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72863671, jitterRate=0.08575330674648285}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:09,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 585751c2808ad67a24a61f19e9ac17f5: Running coprocessor pre-open hook at 1676290929648Writing region info on filesystem at 1676290929648Initializing all the Stores at 1676290929649 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290929649Cleaning up temporary data from old regions at 1676290929670 (+21 ms)Cleaning up detritus from prior splits at 1676290929670Running coprocessor post-open hooks at 1676290929675 (+5 ms)Region opened successfully at 1676290929675 2023-02-13 12:22:09,676 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5., pid=43, masterSystemTime=1676290929642 2023-02-13 12:22:09,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:09,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 50cc7d0236be777dd672124c8ff93b56; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67444558, jitterRate=0.005002230405807495}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:09,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 50cc7d0236be777dd672124c8ff93b56: Running coprocessor pre-open hook at 1676290929653Writing region info on filesystem at 1676290929653Initializing all the Stores at 1676290929654 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290929654Cleaning up temporary data from old regions at 1676290929673 (+19 ms)Cleaning up detritus from prior splits at 1676290929674 (+1 ms)Running coprocessor post-open hooks at 1676290929678 (+4 ms)Region opened successfully at 1676290929678 2023-02-13 12:22:09,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:09,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56., pid=47, masterSystemTime=1676290929648 2023-02-13 12:22:09,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:09,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:09,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:09,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:22:09,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:09,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:09,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6d6fd747f58897a134fa0a460eb9ec99; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73682660, jitterRate=0.09795719385147095}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:09,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:09,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6d6fd747f58897a134fa0a460eb9ec99: Running coprocessor pre-open hook at 1676290929658Writing region info on filesystem at 1676290929658Initializing all the Stores at 1676290929660 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290929660Cleaning up temporary data from old regions at 1676290929674 (+14 ms)Cleaning up detritus from prior splits at 1676290929674Running coprocessor post-open hooks at 1676290929680 (+6 ms)Region opened successfully at 1676290929680 2023-02-13 12:22:09,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:09,681 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:09,681 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99., pid=46, masterSystemTime=1676290929652 2023-02-13 12:22:09,681 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290929681"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290929681"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290929681"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290929681"}]},"ts":"1676290929681"} 2023-02-13 12:22:09,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:09,682 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:09,682 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:09,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:22:09,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:09,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:09,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:09,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:09,683 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:09,683 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290929683"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290929683"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290929683"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290929683"}]},"ts":"1676290929683"} 2023-02-13 12:22:09,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:09,683 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:09,683 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:09,684 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:09,686 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:09,686 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:09,686 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:09,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:22:09,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:09,687 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6e9099611b20e9378ce559b4b3a17188 columnFamilyName cf 2023-02-13 12:22:09,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:09,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:09,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:09,687 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:09,687 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:09,688 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81a0069ff66a08b05e0e5fe4210e0ce2 columnFamilyName cf 2023-02-13 12:22:09,689 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:09,689 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290929689"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290929689"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290929689"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290929689"}]},"ts":"1676290929689"} 2023-02-13 12:22:09,690 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(310): Store=81a0069ff66a08b05e0e5fe4210e0ce2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:09,691 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=38 2023-02-13 12:22:09,691 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=38, state=SUCCESS; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061 in 187 msec 2023-02-13 12:22:09,692 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=42 2023-02-13 12:22:09,692 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=42, state=SUCCESS; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148 in 194 msec 2023-02-13 12:22:09,693 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN in 362 msec 2023-02-13 12:22:09,694 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN in 363 msec 2023-02-13 12:22:09,694 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=39 2023-02-13 12:22:09,694 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=39, state=SUCCESS; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111 in 194 msec 2023-02-13 12:22:09,696 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN in 365 msec 2023-02-13 12:22:09,700 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:09,700 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(310): Store=6e9099611b20e9378ce559b4b3a17188/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:09,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:09,704 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:09,705 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:09,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:09,705 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 197a58e90a6010ab47efab725a975e4c columnFamilyName cf 2023-02-13 12:22:09,706 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(310): Store=197a58e90a6010ab47efab725a975e4c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:09,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:09,709 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:09,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81a0069ff66a08b05e0e5fe4210e0ce2; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70272555, jitterRate=0.047142669558525085}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:09,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:09,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Running coprocessor pre-open hook at 1676290929683Writing region info on filesystem at 1676290929683Initializing all the Stores at 1676290929684 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290929684Cleaning up temporary data from old regions at 1676290929705 (+21 ms)Cleaning up detritus from prior splits at 1676290929706 (+1 ms)Running coprocessor post-open hooks at 1676290929710 (+4 ms)Region opened successfully at 1676290929710 2023-02-13 12:22:09,711 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2., pid=44, masterSystemTime=1676290929648 2023-02-13 12:22:09,713 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:09,713 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:09,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:09,715 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:09,715 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290929714"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290929714"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290929714"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290929714"}]},"ts":"1676290929714"} 2023-02-13 12:22:09,715 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:09,718 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 197a58e90a6010ab47efab725a975e4c; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73177957, jitterRate=0.09043653309345245}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:09,718 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 197a58e90a6010ab47efab725a975e4c: Running coprocessor pre-open hook at 1676290929687Writing region info on filesystem at 1676290929687Initializing all the Stores at 1676290929689 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290929689Cleaning up temporary data from old regions at 1676290929711 (+22 ms)Cleaning up detritus from prior splits at 1676290929711Running coprocessor post-open hooks at 1676290929718 (+7 ms)Region opened successfully at 1676290929718 2023-02-13 12:22:09,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:09,720 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c., pid=48, masterSystemTime=1676290929652 2023-02-13 12:22:09,722 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=40 2023-02-13 12:22:09,722 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=40, state=SUCCESS; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061 in 226 msec 2023-02-13 12:22:09,724 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:09,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:09,725 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:09,725 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290929725"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290929725"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290929725"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290929725"}]},"ts":"1676290929725"} 2023-02-13 12:22:09,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:09,728 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN in 393 msec 2023-02-13 12:22:09,729 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6e9099611b20e9378ce559b4b3a17188; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64828293, jitterRate=-0.03398315608501434}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:09,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6e9099611b20e9378ce559b4b3a17188: Running coprocessor pre-open hook at 1676290929681Writing region info on filesystem at 1676290929681Initializing all the Stores at 1676290929683 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290929683Cleaning up temporary data from old regions at 1676290929721 (+38 ms)Cleaning up detritus from prior splits at 1676290929722 (+1 ms)Running coprocessor post-open hooks at 1676290929729 (+7 ms)Region opened successfully at 1676290929729 2023-02-13 12:22:09,731 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188., pid=45, masterSystemTime=1676290929642 2023-02-13 12:22:09,735 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:09,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:09,735 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=41 2023-02-13 12:22:09,736 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=41, state=SUCCESS; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111 in 225 msec 2023-02-13 12:22:09,736 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:09,736 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290929736"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290929736"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290929736"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290929736"}]},"ts":"1676290929736"} 2023-02-13 12:22:09,741 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN in 407 msec 2023-02-13 12:22:09,743 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=37 2023-02-13 12:22:09,743 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=37, state=SUCCESS; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148 in 246 msec 2023-02-13 12:22:09,747 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-02-13 12:22:09,747 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN in 414 msec 2023-02-13 12:22:09,748 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290929748"}]},"ts":"1676290929748"} 2023-02-13 12:22:09,751 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLED in hbase:meta 2023-02-13 12:22:09,786 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676290926065' was successfully enabled. 2023-02-13 12:22:09,791 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676290926065 in 484 msec 2023-02-13 12:22:10,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 12:22:10,067 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 36 completed 2023-02-13 12:22:10,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:10,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:10,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:10,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:10,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:10,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:10,142 DEBUG [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676290926065 2023-02-13 12:22:10,143 DEBUG [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:10,144 INFO [Listener at localhost.localdomain/46179] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 12:22:10,170 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 12:22:10,174 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 12:22:10,177 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 12:22:10,180 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 12:22:10,183 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 12:22:10,194 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676290926065 2023-02-13 12:22:10,195 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676290926065 2023-02-13 12:22:10,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:10,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 12:22:10,201 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290930201"}]},"ts":"1676290930201"} 2023-02-13 12:22:10,204 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLING in hbase:meta 2023-02-13 12:22:10,216 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676290926065 to state=DISABLING 2023-02-13 12:22:10,217 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN}] 2023-02-13 12:22:10,220 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN 2023-02-13 12:22:10,221 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN 2023-02-13 12:22:10,221 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN 2023-02-13 12:22:10,222 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN 2023-02-13 12:22:10,222 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN 2023-02-13 12:22:10,223 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:10,223 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290930223"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290930223"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290930223"}]},"ts":"1676290930223"} 2023-02-13 12:22:10,224 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:10,224 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:10,224 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290930224"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290930224"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290930224"}]},"ts":"1676290930224"} 2023-02-13 12:22:10,224 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290930224"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290930224"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290930224"}]},"ts":"1676290930224"} 2023-02-13 12:22:10,226 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:10,226 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290930226"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290930226"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290930226"}]},"ts":"1676290930226"} 2023-02-13 12:22:10,226 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:10,226 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290930226"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290930226"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290930226"}]},"ts":"1676290930226"} 2023-02-13 12:22:10,228 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=52, state=RUNNABLE; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:10,232 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=55, state=RUNNABLE; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:10,235 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=51, state=RUNNABLE; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:10,236 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=50, state=RUNNABLE; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:10,237 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN 2023-02-13 12:22:10,239 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=54, state=RUNNABLE; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:10,240 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:10,240 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290930240"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290930240"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290930240"}]},"ts":"1676290930240"} 2023-02-13 12:22:10,243 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=53, state=RUNNABLE; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:10,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:10,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 197a58e90a6010ab47efab725a975e4c, disabling compactions & flushes 2023-02-13 12:22:10,385 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:10,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:10,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. after waiting 0 ms 2023-02-13 12:22:10,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:10,386 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 197a58e90a6010ab47efab725a975e4c 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-13 12:22:10,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:10,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 585751c2808ad67a24a61f19e9ac17f5, disabling compactions & flushes 2023-02-13 12:22:10,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:10,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:10,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. after waiting 0 ms 2023-02-13 12:22:10,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:10,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 585751c2808ad67a24a61f19e9ac17f5 1/1 column families, dataSize=23.35 KB heapSize=50.45 KB 2023-02-13 12:22:10,391 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:10,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 50cc7d0236be777dd672124c8ff93b56, disabling compactions & flushes 2023-02-13 12:22:10,392 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:10,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:10,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. after waiting 0 ms 2023-02-13 12:22:10,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:10,392 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 50cc7d0236be777dd672124c8ff93b56 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-13 12:22:10,447 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 12:22:10,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 12:22:10,614 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-13 12:22:10,615 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676290926065' 2023-02-13 12:22:10,616 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-13 12:22:10,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 2023-02-13 12:22:10,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 2023-02-13 12:22:10,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a 2023-02-13 12:22:10,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/.tmp/cf/24b7ea7f7b1d42cfa460732691ead827 2023-02-13 12:22:10,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/.tmp/cf/3e370cf8b62343469a87e9049f031efb 2023-02-13 12:22:10,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=23.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/.tmp/cf/68333e5f91e3413fbabd03a7c9da49fa 2023-02-13 12:22:10,671 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/.tmp/cf/68333e5f91e3413fbabd03a7c9da49fa as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa 2023-02-13 12:22:10,671 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/.tmp/cf/3e370cf8b62343469a87e9049f031efb as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb 2023-02-13 12:22:10,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/.tmp/cf/24b7ea7f7b1d42cfa460732691ead827 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827 2023-02-13 12:22:10,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa, entries=357, sequenceid=8, filesize=65.7 K 2023-02-13 12:22:10,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827, entries=26, sequenceid=8, filesize=9.3 K 2023-02-13 12:22:10,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb, entries=32, sequenceid=8, filesize=10.4 K 2023-02-13 12:22:10,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 50cc7d0236be777dd672124c8ff93b56 in 291ms, sequenceid=8, compaction requested=false 2023-02-13 12:22:10,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 197a58e90a6010ab47efab725a975e4c in 297ms, sequenceid=8, compaction requested=false 2023-02-13 12:22:10,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~23.35 KB/23915, heapSize ~50.44 KB/51648, currentSize=0 B/0 for 585751c2808ad67a24a61f19e9ac17f5 in 295ms, sequenceid=8, compaction requested=false 2023-02-13 12:22:10,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:22:10,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:22:10,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:10,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 50cc7d0236be777dd672124c8ff93b56: Waiting for close lock at 1676290930391Running coprocessor pre-close hooks at 1676290930391Disabling compacts and flushes for region at 1676290930391Disabling writes for close at 1676290930392 (+1 ms)Obtaining lock to block concurrent updates at 1676290930392Preparing flush snapshotting stores in 50cc7d0236be777dd672124c8ff93b56 at 1676290930392Finished memstore snapshotting testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676290930396 (+4 ms)Flushing stores of testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. at 1676290930398 (+2 ms)Flushing 50cc7d0236be777dd672124c8ff93b56/cf: creating writer at 1676290930401 (+3 ms)Flushing mob file 50cc7d0236be777dd672124c8ff93b56/cf: appending metadata at 1676290930478 (+77 ms)Flushing mob file 50cc7d0236be777dd672124c8ff93b56/cf: closing flushed file at 1676290930478Flushing 50cc7d0236be777dd672124c8ff93b56/cf: appending metadata at 1676290930629 (+151 ms)Flushing 50cc7d0236be777dd672124c8ff93b56/cf: closing flushed file at 1676290930632 (+3 ms)Flushing 50cc7d0236be777dd672124c8ff93b56/cf: reopening flushed file at 1676290930673 (+41 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 50cc7d0236be777dd672124c8ff93b56 in 291ms, sequenceid=8, compaction requested=false at 1676290930683 (+10 ms)Writing region close event to WAL at 1676290930691 (+8 ms)Running coprocessor post-close hooks at 1676290930696 (+5 ms)Closed at 1676290930697 (+1 ms) 2023-02-13 12:22:10,698 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:22:10,698 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:10,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 585751c2808ad67a24a61f19e9ac17f5: Waiting for close lock at 1676290930387Running coprocessor pre-close hooks at 1676290930387Disabling compacts and flushes for region at 1676290930387Disabling writes for close at 1676290930388 (+1 ms)Obtaining lock to block concurrent updates at 1676290930388Preparing flush snapshotting stores in 585751c2808ad67a24a61f19e9ac17f5 at 1676290930388Finished memstore snapshotting testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5., syncing WAL and waiting on mvcc, flushsize=dataSize=23915, getHeapSize=51648, getOffHeapSize=0, getCellsCount=357 at 1676290930396 (+8 ms)Flushing stores of testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. at 1676290930398 (+2 ms)Flushing 585751c2808ad67a24a61f19e9ac17f5/cf: creating writer at 1676290930401 (+3 ms)Flushing mob file 585751c2808ad67a24a61f19e9ac17f5/cf: appending metadata at 1676290930525 (+124 ms)Flushing mob file 585751c2808ad67a24a61f19e9ac17f5/cf: closing flushed file at 1676290930525Flushing 585751c2808ad67a24a61f19e9ac17f5/cf: appending metadata at 1676290930629 (+104 ms)Flushing 585751c2808ad67a24a61f19e9ac17f5/cf: closing flushed file at 1676290930632 (+3 ms)Flushing 585751c2808ad67a24a61f19e9ac17f5/cf: reopening flushed file at 1676290930673 (+41 ms)Finished flush of dataSize ~23.35 KB/23915, heapSize ~50.44 KB/51648, currentSize=0 B/0 for 585751c2808ad67a24a61f19e9ac17f5 in 295ms, sequenceid=8, compaction requested=false at 1676290930684 (+11 ms)Writing region close event to WAL at 1676290930691 (+7 ms)Running coprocessor post-close hooks at 1676290930698 (+7 ms)Closed at 1676290930698 2023-02-13 12:22:10,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:10,700 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 197a58e90a6010ab47efab725a975e4c: Waiting for close lock at 1676290930384Running coprocessor pre-close hooks at 1676290930384Disabling compacts and flushes for region at 1676290930384Disabling writes for close at 1676290930385 (+1 ms)Obtaining lock to block concurrent updates at 1676290930386 (+1 ms)Preparing flush snapshotting stores in 197a58e90a6010ab47efab725a975e4c at 1676290930386Finished memstore snapshotting testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676290930396 (+10 ms)Flushing stores of testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. at 1676290930398 (+2 ms)Flushing 197a58e90a6010ab47efab725a975e4c/cf: creating writer at 1676290930401 (+3 ms)Flushing mob file 197a58e90a6010ab47efab725a975e4c/cf: appending metadata at 1676290930496 (+95 ms)Flushing mob file 197a58e90a6010ab47efab725a975e4c/cf: closing flushed file at 1676290930496Flushing 197a58e90a6010ab47efab725a975e4c/cf: appending metadata at 1676290930629 (+133 ms)Flushing 197a58e90a6010ab47efab725a975e4c/cf: closing flushed file at 1676290930632 (+3 ms)Flushing 197a58e90a6010ab47efab725a975e4c/cf: reopening flushed file at 1676290930673 (+41 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 197a58e90a6010ab47efab725a975e4c in 297ms, sequenceid=8, compaction requested=false at 1676290930684 (+11 ms)Writing region close event to WAL at 1676290930691 (+7 ms)Running coprocessor post-close hooks at 1676290930699 (+8 ms)Closed at 1676290930699 2023-02-13 12:22:10,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:10,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:10,701 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81a0069ff66a08b05e0e5fe4210e0ce2, disabling compactions & flushes 2023-02-13 12:22:10,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:10,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:10,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. after waiting 0 ms 2023-02-13 12:22:10,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:10,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 81a0069ff66a08b05e0e5fe4210e0ce2 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-13 12:22:10,702 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSED 2023-02-13 12:22:10,702 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290930702"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290930702"}]},"ts":"1676290930702"} 2023-02-13 12:22:10,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:10,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:10,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6e9099611b20e9378ce559b4b3a17188, disabling compactions & flushes 2023-02-13 12:22:10,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:10,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:10,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. after waiting 0 ms 2023-02-13 12:22:10,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:10,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 6e9099611b20e9378ce559b4b3a17188 1/1 column families, dataSize=1.50 KB heapSize=3.48 KB 2023-02-13 12:22:10,703 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSED 2023-02-13 12:22:10,704 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290930703"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290930703"}]},"ts":"1676290930703"} 2023-02-13 12:22:10,704 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:10,704 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:10,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6d6fd747f58897a134fa0a460eb9ec99, disabling compactions & flushes 2023-02-13 12:22:10,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:10,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:10,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. after waiting 0 ms 2023-02-13 12:22:10,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:10,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 6d6fd747f58897a134fa0a460eb9ec99 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-13 12:22:10,707 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSED 2023-02-13 12:22:10,707 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290930707"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290930707"}]},"ts":"1676290930707"} 2023-02-13 12:22:10,712 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=51 2023-02-13 12:22:10,712 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=55 2023-02-13 12:22:10,712 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=51, state=SUCCESS; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061 in 471 msec 2023-02-13 12:22:10,712 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=55, state=SUCCESS; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148 in 476 msec 2023-02-13 12:22:10,726 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=54 2023-02-13 12:22:10,726 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN in 495 msec 2023-02-13 12:22:10,726 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=54, state=SUCCESS; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111 in 473 msec 2023-02-13 12:22:10,726 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN in 495 msec 2023-02-13 12:22:10,728 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN in 509 msec 2023-02-13 12:22:10,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c 2023-02-13 12:22:10,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f 2023-02-13 12:22:10,789 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f 2023-02-13 12:22:10,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/.tmp/cf/4446486d315045788982bdebc8a2bb90 2023-02-13 12:22:10,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/.tmp/cf/31b72b71ccea46d8a373f1588b857585 2023-02-13 12:22:10,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/.tmp/cf/0a65255f37e3420c9fdfa034125fc5c8 2023-02-13 12:22:10,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/.tmp/cf/4446486d315045788982bdebc8a2bb90 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90 2023-02-13 12:22:10,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/.tmp/cf/31b72b71ccea46d8a373f1588b857585 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585 2023-02-13 12:22:10,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/.tmp/cf/0a65255f37e3420c9fdfa034125fc5c8 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8 2023-02-13 12:22:10,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90, entries=32, sequenceid=8, filesize=10.4 K 2023-02-13 12:22:10,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585, entries=30, sequenceid=8, filesize=10.0 K 2023-02-13 12:22:10,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8, entries=23, sequenceid=8, filesize=8.8 K 2023-02-13 12:22:10,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 81a0069ff66a08b05e0e5fe4210e0ce2 in 155ms, sequenceid=8, compaction requested=false 2023-02-13 12:22:10,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for 6e9099611b20e9378ce559b4b3a17188 in 154ms, sequenceid=8, compaction requested=false 2023-02-13 12:22:10,860 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 6d6fd747f58897a134fa0a460eb9ec99 in 154ms, sequenceid=8, compaction requested=false 2023-02-13 12:22:10,871 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:22:10,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:22:10,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:22:10,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:10,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6e9099611b20e9378ce559b4b3a17188: Waiting for close lock at 1676290930703Running coprocessor pre-close hooks at 1676290930703Disabling compacts and flushes for region at 1676290930703Disabling writes for close at 1676290930703Obtaining lock to block concurrent updates at 1676290930703Preparing flush snapshotting stores in 6e9099611b20e9378ce559b4b3a17188 at 1676290930703Finished memstore snapshotting testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188., syncing WAL and waiting on mvcc, flushsize=dataSize=1539, getHeapSize=3552, getOffHeapSize=0, getCellsCount=23 at 1676290930703Flushing stores of testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. at 1676290930706 (+3 ms)Flushing 6e9099611b20e9378ce559b4b3a17188/cf: creating writer at 1676290930706Flushing mob file 6e9099611b20e9378ce559b4b3a17188/cf: appending metadata at 1676290930737 (+31 ms)Flushing mob file 6e9099611b20e9378ce559b4b3a17188/cf: closing flushed file at 1676290930737Flushing 6e9099611b20e9378ce559b4b3a17188/cf: appending metadata at 1676290930793 (+56 ms)Flushing 6e9099611b20e9378ce559b4b3a17188/cf: closing flushed file at 1676290930793Flushing 6e9099611b20e9378ce559b4b3a17188/cf: reopening flushed file at 1676290930846 (+53 ms)Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for 6e9099611b20e9378ce559b4b3a17188 in 154ms, sequenceid=8, compaction requested=false at 1676290930857 (+11 ms)Writing region close event to WAL at 1676290930866 (+9 ms)Running coprocessor post-close hooks at 1676290930872 (+6 ms)Closed at 1676290930873 (+1 ms) 2023-02-13 12:22:10,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:10,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:10,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6d6fd747f58897a134fa0a460eb9ec99: Waiting for close lock at 1676290930704Running coprocessor pre-close hooks at 1676290930704Disabling compacts and flushes for region at 1676290930704Disabling writes for close at 1676290930706 (+2 ms)Obtaining lock to block concurrent updates at 1676290930707 (+1 ms)Preparing flush snapshotting stores in 6d6fd747f58897a134fa0a460eb9ec99 at 1676290930707Finished memstore snapshotting testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676290930707Flushing stores of testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. at 1676290930708 (+1 ms)Flushing 6d6fd747f58897a134fa0a460eb9ec99/cf: creating writer at 1676290930709 (+1 ms)Flushing mob file 6d6fd747f58897a134fa0a460eb9ec99/cf: appending metadata at 1676290930746 (+37 ms)Flushing mob file 6d6fd747f58897a134fa0a460eb9ec99/cf: closing flushed file at 1676290930746Flushing 6d6fd747f58897a134fa0a460eb9ec99/cf: appending metadata at 1676290930790 (+44 ms)Flushing 6d6fd747f58897a134fa0a460eb9ec99/cf: closing flushed file at 1676290930790Flushing 6d6fd747f58897a134fa0a460eb9ec99/cf: reopening flushed file at 1676290930842 (+52 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 6d6fd747f58897a134fa0a460eb9ec99 in 154ms, sequenceid=8, compaction requested=false at 1676290930860 (+18 ms)Writing region close event to WAL at 1676290930867 (+7 ms)Running coprocessor post-close hooks at 1676290930873 (+6 ms)Closed at 1676290930874 (+1 ms) 2023-02-13 12:22:10,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Waiting for close lock at 1676290930701Running coprocessor pre-close hooks at 1676290930701Disabling compacts and flushes for region at 1676290930701Disabling writes for close at 1676290930702 (+1 ms)Obtaining lock to block concurrent updates at 1676290930702Preparing flush snapshotting stores in 81a0069ff66a08b05e0e5fe4210e0ce2 at 1676290930702Finished memstore snapshotting testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676290930702Flushing stores of testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. at 1676290930705 (+3 ms)Flushing 81a0069ff66a08b05e0e5fe4210e0ce2/cf: creating writer at 1676290930706 (+1 ms)Flushing mob file 81a0069ff66a08b05e0e5fe4210e0ce2/cf: appending metadata at 1676290930733 (+27 ms)Flushing mob file 81a0069ff66a08b05e0e5fe4210e0ce2/cf: closing flushed file at 1676290930733Flushing 81a0069ff66a08b05e0e5fe4210e0ce2/cf: appending metadata at 1676290930790 (+57 ms)Flushing 81a0069ff66a08b05e0e5fe4210e0ce2/cf: closing flushed file at 1676290930790Flushing 81a0069ff66a08b05e0e5fe4210e0ce2/cf: reopening flushed file at 1676290930845 (+55 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 81a0069ff66a08b05e0e5fe4210e0ce2 in 155ms, sequenceid=8, compaction requested=false at 1676290930857 (+12 ms)Writing region close event to WAL at 1676290930866 (+9 ms)Running coprocessor post-close hooks at 1676290930874 (+8 ms)Closed at 1676290930874 2023-02-13 12:22:10,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:10,883 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSED 2023-02-13 12:22:10,883 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290930883"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290930883"}]},"ts":"1676290930883"} 2023-02-13 12:22:10,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:10,884 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSED 2023-02-13 12:22:10,884 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290930884"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290930884"}]},"ts":"1676290930884"} 2023-02-13 12:22:10,885 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:10,886 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSED 2023-02-13 12:22:10,886 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290930886"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290930886"}]},"ts":"1676290930886"} 2023-02-13 12:22:10,891 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=50 2023-02-13 12:22:10,891 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=50, state=SUCCESS; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148 in 649 msec 2023-02-13 12:22:10,895 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=52 2023-02-13 12:22:10,896 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=52, state=SUCCESS; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111 in 660 msec 2023-02-13 12:22:10,897 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=53 2023-02-13 12:22:10,897 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN in 674 msec 2023-02-13 12:22:10,897 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=53, state=SUCCESS; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061 in 646 msec 2023-02-13 12:22:10,899 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN in 678 msec 2023-02-13 12:22:10,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=53, resume processing ppid=49 2023-02-13 12:22:10,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN in 680 msec 2023-02-13 12:22:10,902 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290930902"}]},"ts":"1676290930902"} 2023-02-13 12:22:10,904 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLED in hbase:meta 2023-02-13 12:22:10,949 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676290926065 to state=DISABLED 2023-02-13 12:22:10,952 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676290926065 in 755 msec 2023-02-13 12:22:10,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 12:22:10,955 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 49 completed 2023-02-13 12:22:10,957 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=snaptb0-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:10,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676290930957 (current time:1676290930957). 2023-02-13 12:22:10,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 12:22:10,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(139): Connect 0x6c88993c to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:11,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a09a7ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:11,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:11,022 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:38276, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:11,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c88993c to 127.0.0.1:62541 2023-02-13 12:22:11,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:11,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 12:22:11,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:11,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:11,029 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:11,030 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE; timeout=600000, timestamp=1676291531030 2023-02-13 12:22:11,030 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676290926065 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676290926065 2023-02-13 12:22:11,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:11,031 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 12:22:11,031 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 12:22:11,047 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } 2023-02-13 12:22:11,049 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:11,049 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:11,049 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:11,049 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:11,050 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:11,050 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:11,049 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:11,050 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:11,050 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:11,050 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:11,050 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:11,050 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:11,050 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:11,050 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:11,054 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb] hfiles 2023-02-13 12:22:11,054 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb 2023-02-13 12:22:11,054 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90] hfiles 2023-02-13 12:22:11,054 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90 2023-02-13 12:22:11,054 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827] hfiles 2023-02-13 12:22:11,054 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827 2023-02-13 12:22:11,054 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585] hfiles 2023-02-13 12:22:11,054 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585 2023-02-13 12:22:11,055 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa] hfiles 2023-02-13 12:22:11,055 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa 2023-02-13 12:22:11,056 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8] hfiles 2023-02-13 12:22:11,056 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8 2023-02-13 12:22:11,058 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f] hfiles 2023-02-13 12:22:11,058 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a 2023-02-13 12:22:11,059 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 2023-02-13 12:22:11,059 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c 2023-02-13 12:22:11,059 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f 2023-02-13 12:22:11,059 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 2023-02-13 12:22:11,059 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f 2023-02-13 12:22:11,103 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } as finished. 2023-02-13 12:22:11,103 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676290926065 2023-02-13 12:22:11,104 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/snaptb0-1676290926065 2023-02-13 12:22:11,151 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/snaptb0-1676290926065 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/snaptb0-1676290926065 2023-02-13 12:22:11,177 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676290926065 of table testRestoreSnapshot-1676290926065 completed 2023-02-13 12:22:11,178 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/snaptb0-1676290926065 2023-02-13 12:22:11,178 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/snaptb0-1676290926065 2023-02-13 12:22:11,178 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676290926065 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676290926065 at 1676290931031Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } at 1676290931048 (+17 ms)Consolidate snapshot: snaptb0-1676290926065 at 1676290931103 (+55 ms)Loading Region manifests for snaptb0-1676290926065 at 1676290931103Writing data manifest for snaptb0-1676290926065 at 1676290931118 (+15 ms)Verifying snapshot: snaptb0-1676290926065 at 1676290931135 (+17 ms)Snapshot snaptb0-1676290926065 of table testRestoreSnapshot-1676290926065 completed at 1676290931177 (+42 ms) 2023-02-13 12:22:11,179 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:11,181 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE in 152 msec 2023-02-13 12:22:11,281 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:22:11,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } is done 2023-02-13 12:22:11,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 12:22:11,284 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676290926065 2023-02-13 12:22:11,284 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testRestoreSnapshot-1676290926065 2023-02-13 12:22:11,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:11,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 12:22:11,288 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676290926065 2023-02-13 12:22:11,288 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290931288"}]},"ts":"1676290931288"} 2023-02-13 12:22:11,290 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLING in hbase:meta 2023-02-13 12:22:11,363 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 12:22:11,363 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN}] 2023-02-13 12:22:11,365 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN 2023-02-13 12:22:11,366 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN 2023-02-13 12:22:11,366 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN 2023-02-13 12:22:11,366 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN 2023-02-13 12:22:11,367 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN 2023-02-13 12:22:11,367 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:11,367 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:11,367 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:11,367 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:11,367 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=true 2023-02-13 12:22:11,367 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=true 2023-02-13 12:22:11,367 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=true 2023-02-13 12:22:11,367 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=true 2023-02-13 12:22:11,368 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:11,368 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=true 2023-02-13 12:22:11,368 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN 2023-02-13 12:22:11,369 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:11,369 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=true 2023-02-13 12:22:11,517 INFO [jenkins-hbase16:46677] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 12:22:11,518 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:11,518 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:11,518 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:11,518 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290931518"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290931518"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290931518"}]},"ts":"1676290931518"} 2023-02-13 12:22:11,518 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:11,518 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290931518"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290931518"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290931518"}]},"ts":"1676290931518"} 2023-02-13 12:22:11,518 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:11,518 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290931518"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290931518"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290931518"}]},"ts":"1676290931518"} 2023-02-13 12:22:11,518 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290931518"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290931518"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290931518"}]},"ts":"1676290931518"} 2023-02-13 12:22:11,518 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290931518"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290931518"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290931518"}]},"ts":"1676290931518"} 2023-02-13 12:22:11,520 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=65, state=RUNNABLE; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:11,521 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=64, state=RUNNABLE; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:11,523 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=69, state=RUNNABLE; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:11,527 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=66, state=RUNNABLE; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:11,528 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=67, state=RUNNABLE; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:11,532 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:11,532 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290931532"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290931532"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290931532"}]},"ts":"1676290931532"} 2023-02-13 12:22:11,537 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=68, state=RUNNABLE; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:11,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 12:22:11,676 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:11,676 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:22:11,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:11,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:11,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:11,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:11,678 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:11,679 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:11,679 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:11,680 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 50cc7d0236be777dd672124c8ff93b56 columnFamilyName cf 2023-02-13 12:22:11,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:11,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:22:11,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:11,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:11,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:11,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:11,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:11,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:22:11,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:11,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:11,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:11,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:11,688 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:11,689 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:11,689 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:11,689 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:11,690 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 197a58e90a6010ab47efab725a975e4c columnFamilyName cf 2023-02-13 12:22:11,690 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:11,690 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb 2023-02-13 12:22:11,690 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:11,691 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 585751c2808ad67a24a61f19e9ac17f5 columnFamilyName cf 2023-02-13 12:22:11,691 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(310): Store=50cc7d0236be777dd672124c8ff93b56/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:11,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:11,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:11,698 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827 2023-02-13 12:22:11,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:11,702 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa 2023-02-13 12:22:11,702 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(310): Store=585751c2808ad67a24a61f19e9ac17f5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:11,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 50cc7d0236be777dd672124c8ff93b56; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75335372, jitterRate=0.1225845217704773}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:11,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 50cc7d0236be777dd672124c8ff93b56: Running coprocessor pre-open hook at 1676290931677Writing region info on filesystem at 1676290931677Initializing all the Stores at 1676290931678 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290931678Cleaning up temporary data from old regions at 1676290931696 (+18 ms)Cleaning up detritus from prior splits at 1676290931697 (+1 ms)Running coprocessor post-open hooks at 1676290931703 (+6 ms)Region opened successfully at 1676290931703 2023-02-13 12:22:11,704 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(310): Store=197a58e90a6010ab47efab725a975e4c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:11,704 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56., pid=70, masterSystemTime=1676290931672 2023-02-13 12:22:11,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:11,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:11,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:11,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:11,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:11,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:11,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:22:11,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:11,708 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:11,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:11,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:11,708 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290931708"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290931708"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290931708"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290931708"}]},"ts":"1676290931708"} 2023-02-13 12:22:11,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:11,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:11,715 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=65 2023-02-13 12:22:11,715 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=65, state=SUCCESS; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061 in 191 msec 2023-02-13 12:22:11,715 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:11,716 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:11,717 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:11,717 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN in 352 msec 2023-02-13 12:22:11,717 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 585751c2808ad67a24a61f19e9ac17f5; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61299319, jitterRate=-0.08656896650791168}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:11,717 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:11,717 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 585751c2808ad67a24a61f19e9ac17f5: Running coprocessor pre-open hook at 1676290931688Writing region info on filesystem at 1676290931688Initializing all the Stores at 1676290931689 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290931689Cleaning up temporary data from old regions at 1676290931708 (+19 ms)Cleaning up detritus from prior splits at 1676290931710 (+2 ms)Running coprocessor post-open hooks at 1676290931717 (+7 ms)Region opened successfully at 1676290931717 2023-02-13 12:22:11,717 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:11,718 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81a0069ff66a08b05e0e5fe4210e0ce2 columnFamilyName cf 2023-02-13 12:22:11,718 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5., pid=72, masterSystemTime=1676290931683 2023-02-13 12:22:11,721 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 197a58e90a6010ab47efab725a975e4c; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63583685, jitterRate=-0.05252926051616669}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:11,721 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 197a58e90a6010ab47efab725a975e4c: Running coprocessor pre-open hook at 1676290931687Writing region info on filesystem at 1676290931687Initializing all the Stores at 1676290931688 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290931688Cleaning up temporary data from old regions at 1676290931712 (+24 ms)Cleaning up detritus from prior splits at 1676290931714 (+2 ms)Running coprocessor post-open hooks at 1676290931721 (+7 ms)Region opened successfully at 1676290931721 2023-02-13 12:22:11,724 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:11,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:11,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:11,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c., pid=75, masterSystemTime=1676290931683 2023-02-13 12:22:11,724 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:22:11,725 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:11,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:11,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:11,725 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290931725"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290931725"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290931725"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290931725"}]},"ts":"1676290931725"} 2023-02-13 12:22:11,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:11,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:11,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:11,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:11,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:11,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:22:11,727 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:11,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:11,728 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290931727"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290931727"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290931727"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290931727"}]},"ts":"1676290931727"} 2023-02-13 12:22:11,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:11,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:11,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:11,730 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=69 2023-02-13 12:22:11,730 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=69, state=SUCCESS; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148 in 204 msec 2023-02-13 12:22:11,732 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=68 2023-02-13 12:22:11,732 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=68, state=SUCCESS; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111 in 194 msec 2023-02-13 12:22:11,732 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN in 367 msec 2023-02-13 12:22:11,733 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:11,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN in 369 msec 2023-02-13 12:22:11,734 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:11,734 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:11,735 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d6fd747f58897a134fa0a460eb9ec99 columnFamilyName cf 2023-02-13 12:22:11,737 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585 2023-02-13 12:22:11,740 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:11,740 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(310): Store=81a0069ff66a08b05e0e5fe4210e0ce2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:11,741 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:11,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:11,741 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:11,742 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6e9099611b20e9378ce559b4b3a17188 columnFamilyName cf 2023-02-13 12:22:11,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:11,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:11,749 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81a0069ff66a08b05e0e5fe4210e0ce2; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70235342, jitterRate=0.04658815264701843}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:11,750 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Running coprocessor pre-open hook at 1676290931710Writing region info on filesystem at 1676290931710Initializing all the Stores at 1676290931715 (+5 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290931715Cleaning up temporary data from old regions at 1676290931744 (+29 ms)Cleaning up detritus from prior splits at 1676290931745 (+1 ms)Running coprocessor post-open hooks at 1676290931750 (+5 ms)Region opened successfully at 1676290931750 2023-02-13 12:22:11,751 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2., pid=74, masterSystemTime=1676290931672 2023-02-13 12:22:11,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:11,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:11,753 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:11,754 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290931753"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290931753"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290931753"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290931753"}]},"ts":"1676290931753"} 2023-02-13 12:22:11,758 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90 2023-02-13 12:22:11,758 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=67 2023-02-13 12:22:11,758 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=67, state=SUCCESS; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061 in 228 msec 2023-02-13 12:22:11,760 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN in 395 msec 2023-02-13 12:22:11,760 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(310): Store=6d6fd747f58897a134fa0a460eb9ec99/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:11,762 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8 2023-02-13 12:22:11,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:11,762 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(310): Store=6e9099611b20e9378ce559b4b3a17188/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:11,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:11,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:11,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:11,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:11,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6d6fd747f58897a134fa0a460eb9ec99; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66762678, jitterRate=-0.005158573389053345}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:11,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6d6fd747f58897a134fa0a460eb9ec99: Running coprocessor pre-open hook at 1676290931728Writing region info on filesystem at 1676290931728Initializing all the Stores at 1676290931732 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290931732Cleaning up temporary data from old regions at 1676290931765 (+33 ms)Cleaning up detritus from prior splits at 1676290931766 (+1 ms)Running coprocessor post-open hooks at 1676290931770 (+4 ms)Region opened successfully at 1676290931770 2023-02-13 12:22:11,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99., pid=73, masterSystemTime=1676290931683 2023-02-13 12:22:11,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:11,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6e9099611b20e9378ce559b4b3a17188; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59750892, jitterRate=-0.10964232683181763}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:11,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6e9099611b20e9378ce559b4b3a17188: Running coprocessor pre-open hook at 1676290931725Writing region info on filesystem at 1676290931725Initializing all the Stores at 1676290931736 (+11 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290931736Cleaning up temporary data from old regions at 1676290931766 (+30 ms)Cleaning up detritus from prior splits at 1676290931767 (+1 ms)Running coprocessor post-open hooks at 1676290931774 (+7 ms)Region opened successfully at 1676290931774 2023-02-13 12:22:11,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188., pid=71, masterSystemTime=1676290931683 2023-02-13 12:22:11,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:11,777 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:11,777 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:11,778 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290931777"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290931777"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290931777"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290931777"}]},"ts":"1676290931777"} 2023-02-13 12:22:11,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:11,778 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:11,778 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:11,779 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290931778"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290931778"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290931778"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290931778"}]},"ts":"1676290931778"} 2023-02-13 12:22:11,782 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=66 2023-02-13 12:22:11,782 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=66, state=SUCCESS; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111 in 253 msec 2023-02-13 12:22:11,783 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=64 2023-02-13 12:22:11,783 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=64, state=SUCCESS; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148 in 260 msec 2023-02-13 12:22:11,784 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN in 419 msec 2023-02-13 12:22:11,785 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-02-13 12:22:11,785 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN in 420 msec 2023-02-13 12:22:11,786 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290931786"}]},"ts":"1676290931786"} 2023-02-13 12:22:11,788 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLED in hbase:meta 2023-02-13 12:22:11,799 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676290926065' was successfully enabled. 2023-02-13 12:22:11,802 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676290926065 in 516 msec 2023-02-13 12:22:12,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 12:22:12,041 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 63 completed 2023-02-13 12:22:12,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:12,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:12,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:12,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:12,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:12,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:12,074 DEBUG [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676290926065 2023-02-13 12:22:12,074 DEBUG [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:12,075 INFO [Listener at localhost.localdomain/46179] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 12:22:12,098 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 12:22:12,099 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-13 12:22:12,299 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 12:22:12,479 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 12:22:12,674 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 12:22:12,841 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 12:22:12,973 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 12:22:14,741 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ReadOnlyZKClient(139): Connect 0x156f10be to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:14,753 DEBUG [Listener at localhost.localdomain/46179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6bbc540, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:14,755 DEBUG [hconnection-0xb5551b3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:14,758 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:38282, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:14,762 DEBUG [hconnection-0xb5551b3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:14,765 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:56826, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:14,870 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 12:22:14,879 DEBUG [hconnection-0xb5551b3-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:14,883 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52014, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:15,059 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 12:22:15,192 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 12:22:15,324 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 12:22:15,426 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 12:22:16,200 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 12:22:17,106 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676290926065 2023-02-13 12:22:17,107 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676290926065 2023-02-13 12:22:17,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:17,111 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290937111"}]},"ts":"1676290937111"} 2023-02-13 12:22:17,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 12:22:17,113 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLING in hbase:meta 2023-02-13 12:22:17,124 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676290926065 to state=DISABLING 2023-02-13 12:22:17,125 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN}] 2023-02-13 12:22:17,128 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN 2023-02-13 12:22:17,128 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN 2023-02-13 12:22:17,128 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN 2023-02-13 12:22:17,128 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN 2023-02-13 12:22:17,129 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN 2023-02-13 12:22:17,130 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:17,130 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:17,130 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:17,130 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290937130"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290937130"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290937130"}]},"ts":"1676290937130"} 2023-02-13 12:22:17,130 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:17,130 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290937129"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290937129"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290937129"}]},"ts":"1676290937129"} 2023-02-13 12:22:17,130 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290937130"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290937130"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290937130"}]},"ts":"1676290937130"} 2023-02-13 12:22:17,130 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:17,130 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290937130"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290937130"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290937130"}]},"ts":"1676290937130"} 2023-02-13 12:22:17,130 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290937130"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290937130"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290937130"}]},"ts":"1676290937130"} 2023-02-13 12:22:17,131 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=82, state=RUNNABLE; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:17,133 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=77, state=RUNNABLE; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:17,135 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=79, state=RUNNABLE; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:17,141 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=78, state=RUNNABLE; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:17,143 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=81, state=RUNNABLE; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:17,145 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN 2023-02-13 12:22:17,147 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:17,147 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290937147"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290937147"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290937147"}]},"ts":"1676290937147"} 2023-02-13 12:22:17,149 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=80, state=RUNNABLE; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:17,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:17,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6e9099611b20e9378ce559b4b3a17188, disabling compactions & flushes 2023-02-13 12:22:17,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:17,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:17,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. after waiting 0 ms 2023-02-13 12:22:17,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:17,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 6e9099611b20e9378ce559b4b3a17188 1/1 column families, dataSize=2.42 KB heapSize=5.45 KB 2023-02-13 12:22:17,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:17,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 197a58e90a6010ab47efab725a975e4c, disabling compactions & flushes 2023-02-13 12:22:17,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:17,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:17,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. after waiting 0 ms 2023-02-13 12:22:17,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:17,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 197a58e90a6010ab47efab725a975e4c 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-13 12:22:17,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:17,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 50cc7d0236be777dd672124c8ff93b56, disabling compactions & flushes 2023-02-13 12:22:17,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:17,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:17,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. after waiting 0 ms 2023-02-13 12:22:17,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:17,298 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 50cc7d0236be777dd672124c8ff93b56 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-13 12:22:17,316 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e 2023-02-13 12:22:17,319 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 2023-02-13 12:22:17,334 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c 2023-02-13 12:22:17,338 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/.tmp/cf/05ca7eda38a34d9c85aaeaa016b874da 2023-02-13 12:22:17,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/.tmp/cf/05ca7eda38a34d9c85aaeaa016b874da as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/05ca7eda38a34d9c85aaeaa016b874da 2023-02-13 12:22:17,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/.tmp/cf/189c775dee9b461ca79ff1ad57032a6e 2023-02-13 12:22:17,346 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/05ca7eda38a34d9c85aaeaa016b874da, entries=29, sequenceid=15, filesize=9.8 K 2023-02-13 12:22:17,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 50cc7d0236be777dd672124c8ff93b56 in 49ms, sequenceid=15, compaction requested=false 2023-02-13 12:22:17,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/.tmp/cf/189c775dee9b461ca79ff1ad57032a6e as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/189c775dee9b461ca79ff1ad57032a6e 2023-02-13 12:22:17,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:22:17,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/189c775dee9b461ca79ff1ad57032a6e, entries=39, sequenceid=15, filesize=11.5 K 2023-02-13 12:22:17,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:17,357 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 50cc7d0236be777dd672124c8ff93b56: Waiting for close lock at 1676290937296Running coprocessor pre-close hooks at 1676290937296Disabling compacts and flushes for region at 1676290937296Disabling writes for close at 1676290937298 (+2 ms)Obtaining lock to block concurrent updates at 1676290937298Preparing flush snapshotting stores in 50cc7d0236be777dd672124c8ff93b56 at 1676290937298Finished memstore snapshotting testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676290937299 (+1 ms)Flushing stores of testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. at 1676290937299Flushing 50cc7d0236be777dd672124c8ff93b56/cf: creating writer at 1676290937300 (+1 ms)Flushing mob file 50cc7d0236be777dd672124c8ff93b56/cf: appending metadata at 1676290937305 (+5 ms)Flushing mob file 50cc7d0236be777dd672124c8ff93b56/cf: closing flushed file at 1676290937305Flushing 50cc7d0236be777dd672124c8ff93b56/cf: appending metadata at 1676290937320 (+15 ms)Flushing 50cc7d0236be777dd672124c8ff93b56/cf: closing flushed file at 1676290937320Flushing 50cc7d0236be777dd672124c8ff93b56/cf: reopening flushed file at 1676290937343 (+23 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 50cc7d0236be777dd672124c8ff93b56 in 49ms, sequenceid=15, compaction requested=false at 1676290937347 (+4 ms)Writing region close event to WAL at 1676290937352 (+5 ms)Running coprocessor post-close hooks at 1676290937356 (+4 ms)Closed at 1676290937357 (+1 ms) 2023-02-13 12:22:17,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 197a58e90a6010ab47efab725a975e4c in 62ms, sequenceid=15, compaction requested=false 2023-02-13 12:22:17,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:17,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:17,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81a0069ff66a08b05e0e5fe4210e0ce2, disabling compactions & flushes 2023-02-13 12:22:17,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:17,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:17,363 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSED 2023-02-13 12:22:17,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 12:22:17,363 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290937363"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290937363"}]},"ts":"1676290937363"} 2023-02-13 12:22:17,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. after waiting 0 ms 2023-02-13 12:22:17,364 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:17,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 81a0069ff66a08b05e0e5fe4210e0ce2 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-13 12:22:17,368 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:22:17,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:17,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 197a58e90a6010ab47efab725a975e4c: Waiting for close lock at 1676290937295Running coprocessor pre-close hooks at 1676290937295Disabling compacts and flushes for region at 1676290937295Disabling writes for close at 1676290937296 (+1 ms)Obtaining lock to block concurrent updates at 1676290937296Preparing flush snapshotting stores in 197a58e90a6010ab47efab725a975e4c at 1676290937296Finished memstore snapshotting testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c., syncing WAL and waiting on mvcc, flushsize=dataSize=2611, getHeapSize=5856, getOffHeapSize=0, getCellsCount=39 at 1676290937296Flushing stores of testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. at 1676290937298 (+2 ms)Flushing 197a58e90a6010ab47efab725a975e4c/cf: creating writer at 1676290937299 (+1 ms)Flushing mob file 197a58e90a6010ab47efab725a975e4c/cf: appending metadata at 1676290937305 (+6 ms)Flushing mob file 197a58e90a6010ab47efab725a975e4c/cf: closing flushed file at 1676290937305Flushing 197a58e90a6010ab47efab725a975e4c/cf: appending metadata at 1676290937336 (+31 ms)Flushing 197a58e90a6010ab47efab725a975e4c/cf: closing flushed file at 1676290937336Flushing 197a58e90a6010ab47efab725a975e4c/cf: reopening flushed file at 1676290937353 (+17 ms)Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 197a58e90a6010ab47efab725a975e4c in 62ms, sequenceid=15, compaction requested=false at 1676290937358 (+5 ms)Writing region close event to WAL at 1676290937362 (+4 ms)Running coprocessor post-close hooks at 1676290937369 (+7 ms)Closed at 1676290937369 2023-02-13 12:22:17,370 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=78 2023-02-13 12:22:17,370 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=78, state=SUCCESS; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061 in 224 msec 2023-02-13 12:22:17,372 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN in 245 msec 2023-02-13 12:22:17,372 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:17,372 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:17,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6d6fd747f58897a134fa0a460eb9ec99, disabling compactions & flushes 2023-02-13 12:22:17,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:17,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:17,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. after waiting 0 ms 2023-02-13 12:22:17,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:17,374 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 6d6fd747f58897a134fa0a460eb9ec99 1/1 column families, dataSize=2.42 KB heapSize=5.45 KB 2023-02-13 12:22:17,374 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSED 2023-02-13 12:22:17,374 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290937374"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290937374"}]},"ts":"1676290937374"} 2023-02-13 12:22:17,380 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=81 2023-02-13 12:22:17,380 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=81, state=SUCCESS; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111 in 234 msec 2023-02-13 12:22:17,382 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN in 255 msec 2023-02-13 12:22:17,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 2023-02-13 12:22:17,406 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af 2023-02-13 12:22:17,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/.tmp/cf/48186611db074ebeab6e0941597e1c9e 2023-02-13 12:22:17,734 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/.tmp/cf/48186611db074ebeab6e0941597e1c9e as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/48186611db074ebeab6e0941597e1c9e 2023-02-13 12:22:17,739 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/48186611db074ebeab6e0941597e1c9e, entries=37, sequenceid=15, filesize=11.2 K 2023-02-13 12:22:17,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for 6e9099611b20e9378ce559b4b3a17188 in 454ms, sequenceid=15, compaction requested=false 2023-02-13 12:22:17,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:22:17,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:17,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6e9099611b20e9378ce559b4b3a17188: Waiting for close lock at 1676290937286Running coprocessor pre-close hooks at 1676290937286Disabling compacts and flushes for region at 1676290937286Disabling writes for close at 1676290937286Obtaining lock to block concurrent updates at 1676290937286Preparing flush snapshotting stores in 6e9099611b20e9378ce559b4b3a17188 at 1676290937286Finished memstore snapshotting testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188., syncing WAL and waiting on mvcc, flushsize=dataSize=2477, getHeapSize=5568, getOffHeapSize=0, getCellsCount=37 at 1676290937286Flushing stores of testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. at 1676290937287 (+1 ms)Flushing 6e9099611b20e9378ce559b4b3a17188/cf: creating writer at 1676290937288 (+1 ms)Flushing mob file 6e9099611b20e9378ce559b4b3a17188/cf: appending metadata at 1676290937293 (+5 ms)Flushing mob file 6e9099611b20e9378ce559b4b3a17188/cf: closing flushed file at 1676290937293Flushing 6e9099611b20e9378ce559b4b3a17188/cf: appending metadata at 1676290937318 (+25 ms)Flushing 6e9099611b20e9378ce559b4b3a17188/cf: closing flushed file at 1676290937318Flushing 6e9099611b20e9378ce559b4b3a17188/cf: reopening flushed file at 1676290937735 (+417 ms)Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for 6e9099611b20e9378ce559b4b3a17188 in 454ms, sequenceid=15, compaction requested=false at 1676290937740 (+5 ms)Writing region close event to WAL at 1676290937746 (+6 ms)Running coprocessor post-close hooks at 1676290937758 (+12 ms)Closed at 1676290937759 (+1 ms) 2023-02-13 12:22:17,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:17,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:17,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 585751c2808ad67a24a61f19e9ac17f5, disabling compactions & flushes 2023-02-13 12:22:17,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:17,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:17,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. after waiting 0 ms 2023-02-13 12:22:17,761 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSED 2023-02-13 12:22:17,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:17,762 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290937761"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290937761"}]},"ts":"1676290937761"} 2023-02-13 12:22:17,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 585751c2808ad67a24a61f19e9ac17f5 1/1 column families, dataSize=21.65 KB heapSize=46.80 KB 2023-02-13 12:22:17,765 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=77 2023-02-13 12:22:17,765 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=77, state=SUCCESS; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148 in 630 msec 2023-02-13 12:22:17,766 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN in 640 msec 2023-02-13 12:22:17,794 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 2023-02-13 12:22:17,807 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=21.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/.tmp/cf/79d6c194793c4877a70ea1a6905cd846 2023-02-13 12:22:17,811 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/.tmp/cf/79d6c194793c4877a70ea1a6905cd846 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/79d6c194793c4877a70ea1a6905cd846 2023-02-13 12:22:17,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/79d6c194793c4877a70ea1a6905cd846, entries=331, sequenceid=15, filesize=61.3 K 2023-02-13 12:22:17,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.65 KB/22173, heapSize ~46.78 KB/47904, currentSize=0 B/0 for 585751c2808ad67a24a61f19e9ac17f5 in 54ms, sequenceid=15, compaction requested=false 2023-02-13 12:22:17,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/.tmp/cf/84c9c4a754f148fc8187a50ff196c186 2023-02-13 12:22:17,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/.tmp/cf/d1029a30e4964e55bcbe8195fe1e7315 2023-02-13 12:22:17,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:22:17,827 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:17,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 585751c2808ad67a24a61f19e9ac17f5: Waiting for close lock at 1676290937761Running coprocessor pre-close hooks at 1676290937761Disabling compacts and flushes for region at 1676290937761Disabling writes for close at 1676290937761Obtaining lock to block concurrent updates at 1676290937762 (+1 ms)Preparing flush snapshotting stores in 585751c2808ad67a24a61f19e9ac17f5 at 1676290937762Finished memstore snapshotting testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5., syncing WAL and waiting on mvcc, flushsize=dataSize=22173, getHeapSize=47904, getOffHeapSize=0, getCellsCount=331 at 1676290937762Flushing stores of testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. at 1676290937772 (+10 ms)Flushing 585751c2808ad67a24a61f19e9ac17f5/cf: creating writer at 1676290937773 (+1 ms)Flushing mob file 585751c2808ad67a24a61f19e9ac17f5/cf: appending metadata at 1676290937780 (+7 ms)Flushing mob file 585751c2808ad67a24a61f19e9ac17f5/cf: closing flushed file at 1676290937780Flushing 585751c2808ad67a24a61f19e9ac17f5/cf: appending metadata at 1676290937795 (+15 ms)Flushing 585751c2808ad67a24a61f19e9ac17f5/cf: closing flushed file at 1676290937795Flushing 585751c2808ad67a24a61f19e9ac17f5/cf: reopening flushed file at 1676290937811 (+16 ms)Finished flush of dataSize ~21.65 KB/22173, heapSize ~46.78 KB/47904, currentSize=0 B/0 for 585751c2808ad67a24a61f19e9ac17f5 in 54ms, sequenceid=15, compaction requested=false at 1676290937816 (+5 ms)Writing region close event to WAL at 1676290937820 (+4 ms)Running coprocessor post-close hooks at 1676290937826 (+6 ms)Closed at 1676290937826 2023-02-13 12:22:17,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:17,829 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/.tmp/cf/84c9c4a754f148fc8187a50ff196c186 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/84c9c4a754f148fc8187a50ff196c186 2023-02-13 12:22:17,829 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/.tmp/cf/d1029a30e4964e55bcbe8195fe1e7315 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/d1029a30e4964e55bcbe8195fe1e7315 2023-02-13 12:22:17,830 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSED 2023-02-13 12:22:17,830 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290937830"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290937830"}]},"ts":"1676290937830"} 2023-02-13 12:22:17,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/84c9c4a754f148fc8187a50ff196c186, entries=27, sequenceid=15, filesize=9.5 K 2023-02-13 12:22:17,839 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=82 2023-02-13 12:22:17,839 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=82, state=SUCCESS; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148 in 701 msec 2023-02-13 12:22:17,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 81a0069ff66a08b05e0e5fe4210e0ce2 in 475ms, sequenceid=15, compaction requested=false 2023-02-13 12:22:17,840 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/d1029a30e4964e55bcbe8195fe1e7315, entries=37, sequenceid=15, filesize=11.2 K 2023-02-13 12:22:17,841 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN in 714 msec 2023-02-13 12:22:17,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for 6d6fd747f58897a134fa0a460eb9ec99 in 468ms, sequenceid=15, compaction requested=false 2023-02-13 12:22:17,856 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:22:17,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:22:17,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:17,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:17,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6d6fd747f58897a134fa0a460eb9ec99: Waiting for close lock at 1676290937372Running coprocessor pre-close hooks at 1676290937372Disabling compacts and flushes for region at 1676290937372Disabling writes for close at 1676290937373 (+1 ms)Obtaining lock to block concurrent updates at 1676290937374 (+1 ms)Preparing flush snapshotting stores in 6d6fd747f58897a134fa0a460eb9ec99 at 1676290937374Finished memstore snapshotting testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99., syncing WAL and waiting on mvcc, flushsize=dataSize=2477, getHeapSize=5568, getOffHeapSize=0, getCellsCount=37 at 1676290937374Flushing stores of testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. at 1676290937375 (+1 ms)Flushing 6d6fd747f58897a134fa0a460eb9ec99/cf: creating writer at 1676290937375Flushing mob file 6d6fd747f58897a134fa0a460eb9ec99/cf: appending metadata at 1676290937385 (+10 ms)Flushing mob file 6d6fd747f58897a134fa0a460eb9ec99/cf: closing flushed file at 1676290937385Flushing 6d6fd747f58897a134fa0a460eb9ec99/cf: appending metadata at 1676290937407 (+22 ms)Flushing 6d6fd747f58897a134fa0a460eb9ec99/cf: closing flushed file at 1676290937407Flushing 6d6fd747f58897a134fa0a460eb9ec99/cf: reopening flushed file at 1676290937831 (+424 ms)Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for 6d6fd747f58897a134fa0a460eb9ec99 in 468ms, sequenceid=15, compaction requested=false at 1676290937841 (+10 ms)Writing region close event to WAL at 1676290937848 (+7 ms)Running coprocessor post-close hooks at 1676290937857 (+9 ms)Closed at 1676290937858 (+1 ms) 2023-02-13 12:22:17,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Waiting for close lock at 1676290937362Running coprocessor pre-close hooks at 1676290937362Disabling compacts and flushes for region at 1676290937362Disabling writes for close at 1676290937363 (+1 ms)Obtaining lock to block concurrent updates at 1676290937364 (+1 ms)Preparing flush snapshotting stores in 81a0069ff66a08b05e0e5fe4210e0ce2 at 1676290937364Finished memstore snapshotting testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676290937364Flushing stores of testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. at 1676290937364Flushing 81a0069ff66a08b05e0e5fe4210e0ce2/cf: creating writer at 1676290937365 (+1 ms)Flushing mob file 81a0069ff66a08b05e0e5fe4210e0ce2/cf: appending metadata at 1676290937374 (+9 ms)Flushing mob file 81a0069ff66a08b05e0e5fe4210e0ce2/cf: closing flushed file at 1676290937374Flushing 81a0069ff66a08b05e0e5fe4210e0ce2/cf: appending metadata at 1676290937406 (+32 ms)Flushing 81a0069ff66a08b05e0e5fe4210e0ce2/cf: closing flushed file at 1676290937406Flushing 81a0069ff66a08b05e0e5fe4210e0ce2/cf: reopening flushed file at 1676290937830 (+424 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 81a0069ff66a08b05e0e5fe4210e0ce2 in 475ms, sequenceid=15, compaction requested=false at 1676290937839 (+9 ms)Writing region close event to WAL at 1676290937848 (+9 ms)Running coprocessor post-close hooks at 1676290937858 (+10 ms)Closed at 1676290937859 (+1 ms) 2023-02-13 12:22:17,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:17,861 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSED 2023-02-13 12:22:17,862 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290937861"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290937861"}]},"ts":"1676290937861"} 2023-02-13 12:22:17,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:17,863 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSED 2023-02-13 12:22:17,863 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290937863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290937863"}]},"ts":"1676290937863"} 2023-02-13 12:22:17,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 12:22:17,866 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=79 2023-02-13 12:22:17,866 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=79, state=SUCCESS; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111 in 728 msec 2023-02-13 12:22:17,867 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=80 2023-02-13 12:22:17,867 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=80, state=SUCCESS; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061 in 716 msec 2023-02-13 12:22:17,868 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN in 741 msec 2023-02-13 12:22:17,869 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=80, resume processing ppid=76 2023-02-13 12:22:17,869 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN in 742 msec 2023-02-13 12:22:17,870 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290937870"}]},"ts":"1676290937870"} 2023-02-13 12:22:17,872 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLED in hbase:meta 2023-02-13 12:22:17,882 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676290926065 to state=DISABLED 2023-02-13 12:22:17,884 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676290926065 in 776 msec 2023-02-13 12:22:18,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 12:22:18,617 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 76 completed 2023-02-13 12:22:18,618 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=snaptb1-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:18,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676290938618 (current time:1676290938618). 2023-02-13 12:22:18,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 12:22:18,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(139): Connect 0x3f63d88e to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:18,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@170af727, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:18,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:18,685 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:38294, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:18,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f63d88e to 127.0.0.1:62541 2023-02-13 12:22:18,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:18,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 12:22:18,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:18,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:18,692 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:18,693 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE; timeout=600000, timestamp=1676291538693 2023-02-13 12:22:18,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:18,693 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676290926065 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676290926065 2023-02-13 12:22:18,694 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 12:22:18,694 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 12:22:18,711 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } 2023-02-13 12:22:18,713 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:18,713 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:18,714 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:18,714 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:18,714 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:18,714 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:18,714 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:18,714 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:18,716 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:18,714 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:18,716 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:18,715 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:18,715 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:18,722 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/189c775dee9b461ca79ff1ad57032a6e, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827] hfiles 2023-02-13 12:22:18,715 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:18,722 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/189c775dee9b461ca79ff1ad57032a6e 2023-02-13 12:22:18,722 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827 2023-02-13 12:22:18,724 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/84c9c4a754f148fc8187a50ff196c186] hfiles 2023-02-13 12:22:18,726 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585 2023-02-13 12:22:18,726 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/84c9c4a754f148fc8187a50ff196c186 2023-02-13 12:22:18,725 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/05ca7eda38a34d9c85aaeaa016b874da, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb] hfiles 2023-02-13 12:22:18,726 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/05ca7eda38a34d9c85aaeaa016b874da 2023-02-13 12:22:18,726 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb 2023-02-13 12:22:18,727 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/d1029a30e4964e55bcbe8195fe1e7315] hfiles 2023-02-13 12:22:18,727 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90 2023-02-13 12:22:18,727 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/d1029a30e4964e55bcbe8195fe1e7315 2023-02-13 12:22:18,729 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/79d6c194793c4877a70ea1a6905cd846] hfiles 2023-02-13 12:22:18,729 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa 2023-02-13 12:22:18,729 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/79d6c194793c4877a70ea1a6905cd846 2023-02-13 12:22:18,734 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/48186611db074ebeab6e0941597e1c9e] hfiles 2023-02-13 12:22:18,734 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8 2023-02-13 12:22:18,734 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/48186611db074ebeab6e0941597e1c9e 2023-02-13 12:22:18,793 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x156f10be to 127.0.0.1:62541 2023-02-13 12:22:18,794 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:18,804 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f] hfiles 2023-02-13 12:22:18,804 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a 2023-02-13 12:22:18,804 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c 2023-02-13 12:22:18,804 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 2023-02-13 12:22:18,804 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 2023-02-13 12:22:18,804 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c 2023-02-13 12:22:18,804 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af 2023-02-13 12:22:18,804 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e 2023-02-13 12:22:18,805 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f 2023-02-13 12:22:18,805 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 2023-02-13 12:22:18,805 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 2023-02-13 12:22:18,805 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 2023-02-13 12:22:18,805 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f 2023-02-13 12:22:18,849 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } as finished. 2023-02-13 12:22:18,849 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676290926065 2023-02-13 12:22:18,852 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/snaptb1-1676290926065 2023-02-13 12:22:18,904 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/snaptb1-1676290926065 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/snaptb1-1676290926065 2023-02-13 12:22:18,936 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676290926065 of table testRestoreSnapshot-1676290926065 completed 2023-02-13 12:22:18,936 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/snaptb1-1676290926065 2023-02-13 12:22:18,936 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/snaptb1-1676290926065 2023-02-13 12:22:18,936 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676290926065 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676290926065 at 1676290938693Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } at 1676290938711 (+18 ms)Consolidate snapshot: snaptb1-1676290926065 at 1676290938849 (+138 ms)Loading Region manifests for snaptb1-1676290926065 at 1676290938850 (+1 ms)Writing data manifest for snaptb1-1676290926065 at 1676290938862 (+12 ms)Verifying snapshot: snaptb1-1676290926065 at 1676290938876 (+14 ms)Snapshot snaptb1-1676290926065 of table testRestoreSnapshot-1676290926065 completed at 1676290938936 (+60 ms) 2023-02-13 12:22:18,938 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:18,939 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE in 250 msec 2023-02-13 12:22:18,944 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:22:18,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } is done 2023-02-13 12:22:18,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676290926065 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 12:22:18,962 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676290926065-1676290938962 2023-02-13 12:22:18,963 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676290926065-1676290938962 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:18,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676290938963 (current time:1676290938963). 2023-02-13 12:22:18,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 12:22:18,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(139): Connect 0x6f99e13e to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:18,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38c92551, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:18,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:18,981 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:38310, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:18,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f99e13e to 127.0.0.1:62541 2023-02-13 12:22:18,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:18,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 12:22:18,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676290926065-1676290938962 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:18,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:18,986 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:18,987 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE; timeout=600000, timestamp=1676291538987 2023-02-13 12:22:18,987 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676290926065-1676290938962 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676290926065 2023-02-13 12:22:18,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676290926065-1676290938962 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:18,987 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676290926065-1676290938962 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 12:22:18,988 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 12:22:19,001 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676290926065-1676290938962 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } 2023-02-13 12:22:19,002 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:19,002 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:19,003 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:19,003 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:19,003 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:19,003 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:19,003 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:19,003 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:19,003 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:19,003 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:19,003 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:19,003 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:19,003 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:19,003 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/189c775dee9b461ca79ff1ad57032a6e, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827] hfiles 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/189c775dee9b461ca79ff1ad57032a6e 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/48186611db074ebeab6e0941597e1c9e] hfiles 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/84c9c4a754f148fc8187a50ff196c186] hfiles 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/79d6c194793c4877a70ea1a6905cd846] hfiles 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/79d6c194793c4877a70ea1a6905cd846 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/48186611db074ebeab6e0941597e1c9e 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/05ca7eda38a34d9c85aaeaa016b874da, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb] hfiles 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/d1029a30e4964e55bcbe8195fe1e7315] hfiles 2023-02-13 12:22:19,006 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/05ca7eda38a34d9c85aaeaa016b874da 2023-02-13 12:22:19,005 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/84c9c4a754f148fc8187a50ff196c186 2023-02-13 12:22:19,006 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb 2023-02-13 12:22:19,006 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90 2023-02-13 12:22:19,006 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/d1029a30e4964e55bcbe8195fe1e7315 2023-02-13 12:22:19,011 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f] hfiles 2023-02-13 12:22:19,011 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a 2023-02-13 12:22:19,011 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c 2023-02-13 12:22:19,011 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 2023-02-13 12:22:19,011 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 2023-02-13 12:22:19,011 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c 2023-02-13 12:22:19,011 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af 2023-02-13 12:22:19,011 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e 2023-02-13 12:22:19,011 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f 2023-02-13 12:22:19,012 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 2023-02-13 12:22:19,012 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 2023-02-13 12:22:19,012 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 2023-02-13 12:22:19,012 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f 2023-02-13 12:22:19,238 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:22:19,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676290926065-1676290938962 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } is done 2023-02-13 12:22:19,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snaptb0-1676290926065-1676290938962 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 12:22:19,241 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 12:22:19,419 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676290926065-1676290938962 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } as finished. 2023-02-13 12:22:19,419 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676290926065-1676290938962 2023-02-13 12:22:19,420 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676290926065-1676290938962 2023-02-13 12:22:19,473 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676290926065-1676290938962 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/hbase-failsafe-snaptb0-1676290926065-1676290938962 2023-02-13 12:22:19,493 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676290926065-1676290938962 of table testRestoreSnapshot-1676290926065 completed 2023-02-13 12:22:19,493 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676290926065-1676290938962 2023-02-13 12:22:19,493 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676290926065-1676290938962 2023-02-13 12:22:19,494 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676290926065-1676290938962 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676290926065 at 1676290938987Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676290926065-1676290938962 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } at 1676290939001 (+14 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676290926065-1676290938962 at 1676290939419 (+418 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676290926065-1676290938962 at 1676290939419Writing data manifest for hbase-failsafe-snaptb0-1676290926065-1676290938962 at 1676290939428 (+9 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676290926065-1676290938962 at 1676290939460 (+32 ms)Snapshot hbase-failsafe-snaptb0-1676290926065-1676290938962 of table testRestoreSnapshot-1676290926065 completed at 1676290939493 (+33 ms) 2023-02-13 12:22:19,495 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:19,497 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE in 511 msec 2023-02-13 12:22:19,741 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:22:19,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676290926065-1676290938962 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } is done 2023-02-13 12:22:19,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676290926065-1676290938962 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 12:22:19,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676290926065 snapshot=name: "snaptb0-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290930957 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 12:22:19,770 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676290926065 as table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:19,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-13 12:22:19,805 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/.tabledesc/.tableinfo.0000000002 2023-02-13 12:22:19,806 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/.tabledesc/.tableinfo.0000000001 2023-02-13 12:22:19,806 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/.tabledesc/.tableinfo.0000000002 2023-02-13 12:22:19,806 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } 2023-02-13 12:22:19,810 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290930957 type: DISABLED version: 2 ttl: 0 2023-02-13 12:22:19,811 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065 2023-02-13 12:22:19,821 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:19,830 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 2023-02-13 12:22:19,833 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af 2023-02-13 12:22:19,835 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e 2023-02-13 12:22:19,837 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 2023-02-13 12:22:19,839 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 2023-02-13 12:22:19,840 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c 2023-02-13 12:22:19,841 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:19,841 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:19,841 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:19,841 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:19,841 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:19,841 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:19,853 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/189c775dee9b461ca79ff1ad57032a6e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/189c775dee9b461ca79ff1ad57032a6e 2023-02-13 12:22:19,854 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/79d6c194793c4877a70ea1a6905cd846 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/79d6c194793c4877a70ea1a6905cd846 2023-02-13 12:22:19,854 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/05ca7eda38a34d9c85aaeaa016b874da to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/05ca7eda38a34d9c85aaeaa016b874da 2023-02-13 12:22:19,855 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/84c9c4a754f148fc8187a50ff196c186 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/84c9c4a754f148fc8187a50ff196c186 2023-02-13 12:22:19,855 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/d1029a30e4964e55bcbe8195fe1e7315 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/d1029a30e4964e55bcbe8195fe1e7315 2023-02-13 12:22:19,855 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/48186611db074ebeab6e0941597e1c9e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/48186611db074ebeab6e0941597e1c9e 2023-02-13 12:22:19,856 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290930957 type: DISABLED version: 2 ttl: 0 2023-02-13 12:22:19,862 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290939862"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:19,862 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290939862"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:19,862 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290939862"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:19,863 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290939862"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:19,863 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290939862"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:19,863 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290939862"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:19,866 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 12:22:19,866 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-13 12:22:19,867 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290939863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290939863"}]},"ts":"1676290939863"} 2023-02-13 12:22:19,867 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290939863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290939863"}]},"ts":"1676290939863"} 2023-02-13 12:22:19,867 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290939863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290939863"}]},"ts":"1676290939863"} 2023-02-13 12:22:19,867 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290939863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290939863"}]},"ts":"1676290939863"} 2023-02-13 12:22:19,867 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290939863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290939863"}]},"ts":"1676290939863"} 2023-02-13 12:22:19,867 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290939863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290939863"}]},"ts":"1676290939863"} 2023-02-13 12:22:19,870 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 12:22:19,870 INFO [PEWorker-4] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 12:22:19,870 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-13 12:22:19,871 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676290926065 completed! 2023-02-13 12:22:19,874 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676290926065 snapshot=name: "snaptb0-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290930957 type: DISABLED version: 2 ttl: 0 ) in 111 msec 2023-02-13 12:22:20,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-13 12:22:20,026 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676290926065, procId: 91 completed 2023-02-13 12:22:20,027 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676290926065-1676290938962 2023-02-13 12:22:20,031 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "hbase-failsafe-snaptb0-1676290926065-1676290938962" 2023-02-13 12:22:20,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676290926065-1676290938962 2023-02-13 12:22:20,036 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676290926065 2023-02-13 12:22:20,037 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testRestoreSnapshot-1676290926065 2023-02-13 12:22:20,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:20,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 12:22:20,045 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676290926065 2023-02-13 12:22:20,045 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290940045"}]},"ts":"1676290940045"} 2023-02-13 12:22:20,047 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLING in hbase:meta 2023-02-13 12:22:20,078 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 12:22:20,081 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN}] 2023-02-13 12:22:20,083 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN 2023-02-13 12:22:20,083 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN 2023-02-13 12:22:20,083 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN 2023-02-13 12:22:20,083 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN 2023-02-13 12:22:20,084 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:20,085 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:20,085 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:20,085 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:20,085 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN 2023-02-13 12:22:20,086 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN 2023-02-13 12:22:20,088 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:20,088 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:20,234 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 12:22:20,235 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 12:22:20,235 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 12:22:20,235 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 12:22:20,235 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 12:22:20,239 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:20,239 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:20,239 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:20,239 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:20,239 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:20,239 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290940238"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290940238"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290940238"}]},"ts":"1676290940238"} 2023-02-13 12:22:20,239 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290940238"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290940238"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290940238"}]},"ts":"1676290940238"} 2023-02-13 12:22:20,239 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290940238"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290940238"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290940238"}]},"ts":"1676290940238"} 2023-02-13 12:22:20,239 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290940238"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290940238"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290940238"}]},"ts":"1676290940238"} 2023-02-13 12:22:20,239 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290940238"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290940238"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290940238"}]},"ts":"1676290940238"} 2023-02-13 12:22:20,241 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=93, state=RUNNABLE; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:20,243 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=98, state=RUNNABLE; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:20,243 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=96, state=RUNNABLE; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:20,244 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=97, state=RUNNABLE; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:20,246 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:20,246 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=94, state=RUNNABLE; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:20,246 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290940246"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290940246"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290940246"}]},"ts":"1676290940246"} 2023-02-13 12:22:20,248 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=95, state=RUNNABLE; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:20,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 12:22:20,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:20,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:22:20,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:20,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:20,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:22:20,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:20,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:20,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:20,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:20,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:20,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:20,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:20,398 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:20,399 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:20,399 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:20,399 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:20,400 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6e9099611b20e9378ce559b4b3a17188 columnFamilyName cf 2023-02-13 12:22:20,400 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:20,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:22:20,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:20,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:20,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:20,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:20,402 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:20,403 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:20,403 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:20,403 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 197a58e90a6010ab47efab725a975e4c columnFamilyName cf 2023-02-13 12:22:20,405 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8 2023-02-13 12:22:20,406 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(310): Store=6e9099611b20e9378ce559b4b3a17188/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:20,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:20,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:20,408 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:20,408 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:20,409 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d6fd747f58897a134fa0a460eb9ec99 columnFamilyName cf 2023-02-13 12:22:20,410 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827 2023-02-13 12:22:20,410 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(310): Store=197a58e90a6010ab47efab725a975e4c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:20,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:20,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:20,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:20,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:20,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6e9099611b20e9378ce559b4b3a17188; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66518205, jitterRate=-0.008801504969596863}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:20,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6e9099611b20e9378ce559b4b3a17188: Running coprocessor pre-open hook at 1676290940398Writing region info on filesystem at 1676290940398Initializing all the Stores at 1676290940398Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290940398Cleaning up temporary data from old regions at 1676290940408 (+10 ms)Cleaning up detritus from prior splits at 1676290940412 (+4 ms)Running coprocessor post-open hooks at 1676290940415 (+3 ms)Region opened successfully at 1676290940415 2023-02-13 12:22:20,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 197a58e90a6010ab47efab725a975e4c; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58918268, jitterRate=-0.12204939126968384}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:20,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 197a58e90a6010ab47efab725a975e4c: Running coprocessor pre-open hook at 1676290940401Writing region info on filesystem at 1676290940401Initializing all the Stores at 1676290940402 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290940402Cleaning up temporary data from old regions at 1676290940412 (+10 ms)Cleaning up detritus from prior splits at 1676290940413 (+1 ms)Running coprocessor post-open hooks at 1676290940416 (+3 ms)Region opened successfully at 1676290940416 2023-02-13 12:22:20,416 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c., pid=102, masterSystemTime=1676290940397 2023-02-13 12:22:20,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188., pid=99, masterSystemTime=1676290940394 2023-02-13 12:22:20,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:20,418 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:20,418 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:20,418 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:20,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:22:20,418 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290940418"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290940418"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290940418"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290940418"}]},"ts":"1676290940418"} 2023-02-13 12:22:20,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:20,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:20,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:20,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:20,419 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90 2023-02-13 12:22:20,419 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(310): Store=6d6fd747f58897a134fa0a460eb9ec99/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:20,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:20,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:20,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:20,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:22:20,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:20,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:20,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:20,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:20,421 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:20,422 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:20,422 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:20,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:20,422 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:20,422 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290940422"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290940422"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290940422"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290940422"}]},"ts":"1676290940422"} 2023-02-13 12:22:20,422 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 50cc7d0236be777dd672124c8ff93b56 columnFamilyName cf 2023-02-13 12:22:20,423 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:20,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:20,424 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:20,424 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:20,425 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81a0069ff66a08b05e0e5fe4210e0ce2 columnFamilyName cf 2023-02-13 12:22:20,426 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=97 2023-02-13 12:22:20,426 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=97, state=SUCCESS; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,36049,1676290923148 in 176 msec 2023-02-13 12:22:20,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:20,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6d6fd747f58897a134fa0a460eb9ec99; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68489345, jitterRate=0.020570769906044006}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:20,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6d6fd747f58897a134fa0a460eb9ec99: Running coprocessor pre-open hook at 1676290940398Writing region info on filesystem at 1676290940398Initializing all the Stores at 1676290940398Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290940398Cleaning up temporary data from old regions at 1676290940424 (+26 ms)Cleaning up detritus from prior splits at 1676290940425 (+1 ms)Running coprocessor post-open hooks at 1676290940427 (+2 ms)Region opened successfully at 1676290940427 2023-02-13 12:22:20,431 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb 2023-02-13 12:22:20,431 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(310): Store=50cc7d0236be777dd672124c8ff93b56/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:20,432 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585 2023-02-13 12:22:20,432 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(310): Store=81a0069ff66a08b05e0e5fe4210e0ce2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:20,432 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99., pid=104, masterSystemTime=1676290940395 2023-02-13 12:22:20,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:20,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:20,434 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN in 346 msec 2023-02-13 12:22:20,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:20,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:20,438 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=93 2023-02-13 12:22:20,438 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=93, state=SUCCESS; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,34139,1676290923061 in 184 msec 2023-02-13 12:22:20,438 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:20,438 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:20,438 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:20,438 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:22:20,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:20,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:20,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:20,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:20,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:20,440 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:20,441 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290940440"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290940440"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290940440"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290940440"}]},"ts":"1676290940440"} 2023-02-13 12:22:20,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81a0069ff66a08b05e0e5fe4210e0ce2; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66150089, jitterRate=-0.014286860823631287}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:20,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Running coprocessor pre-open hook at 1676290940420Writing region info on filesystem at 1676290940420Initializing all the Stores at 1676290940422 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290940422Cleaning up temporary data from old regions at 1676290940436 (+14 ms)Cleaning up detritus from prior splits at 1676290940437 (+1 ms)Running coprocessor post-open hooks at 1676290940441 (+4 ms)Region opened successfully at 1676290940441 2023-02-13 12:22:20,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:20,443 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN in 360 msec 2023-02-13 12:22:20,443 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2., pid=101, masterSystemTime=1676290940394 2023-02-13 12:22:20,443 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 50cc7d0236be777dd672124c8ff93b56; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63138441, jitterRate=-0.05916391313076019}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:20,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 50cc7d0236be777dd672124c8ff93b56: Running coprocessor pre-open hook at 1676290940419Writing region info on filesystem at 1676290940419Initializing all the Stores at 1676290940420 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290940420Cleaning up temporary data from old regions at 1676290940435 (+15 ms)Cleaning up detritus from prior splits at 1676290940436 (+1 ms)Running coprocessor post-open hooks at 1676290940443 (+7 ms)Region opened successfully at 1676290940443 2023-02-13 12:22:20,444 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56., pid=103, masterSystemTime=1676290940397 2023-02-13 12:22:20,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:20,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:20,446 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=95 2023-02-13 12:22:20,446 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=95, state=SUCCESS; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111 in 196 msec 2023-02-13 12:22:20,446 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:20,446 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290940446"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290940446"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290940446"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290940446"}]},"ts":"1676290940446"} 2023-02-13 12:22:20,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:20,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:20,447 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:20,447 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290940447"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290940447"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290940447"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290940447"}]},"ts":"1676290940447"} 2023-02-13 12:22:20,448 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:20,449 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN in 366 msec 2023-02-13 12:22:20,450 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:20,451 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:20,451 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 585751c2808ad67a24a61f19e9ac17f5 columnFamilyName cf 2023-02-13 12:22:20,452 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=96 2023-02-13 12:22:20,453 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=96, state=SUCCESS; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061 in 206 msec 2023-02-13 12:22:20,457 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=94 2023-02-13 12:22:20,457 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN in 373 msec 2023-02-13 12:22:20,457 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=94, state=SUCCESS; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,36049,1676290923148 in 205 msec 2023-02-13 12:22:20,458 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN in 379 msec 2023-02-13 12:22:20,460 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa 2023-02-13 12:22:20,460 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(310): Store=585751c2808ad67a24a61f19e9ac17f5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:20,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:20,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:20,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:20,467 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 585751c2808ad67a24a61f19e9ac17f5; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74692076, jitterRate=0.11299866437911987}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:20,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 585751c2808ad67a24a61f19e9ac17f5: Running coprocessor pre-open hook at 1676290940439Writing region info on filesystem at 1676290940439Initializing all the Stores at 1676290940440 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290940440Cleaning up temporary data from old regions at 1676290940463 (+23 ms)Cleaning up detritus from prior splits at 1676290940463Running coprocessor post-open hooks at 1676290940468 (+5 ms)Region opened successfully at 1676290940468 2023-02-13 12:22:20,472 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5., pid=100, masterSystemTime=1676290940395 2023-02-13 12:22:20,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:20,474 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:20,474 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:20,474 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290940474"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290940474"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290940474"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290940474"}]},"ts":"1676290940474"} 2023-02-13 12:22:20,477 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=98 2023-02-13 12:22:20,478 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=98, state=SUCCESS; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,34079,1676290923111 in 233 msec 2023-02-13 12:22:20,479 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=98, resume processing ppid=92 2023-02-13 12:22:20,479 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN in 398 msec 2023-02-13 12:22:20,479 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290940479"}]},"ts":"1676290940479"} 2023-02-13 12:22:20,480 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLED in hbase:meta 2023-02-13 12:22:20,510 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676290926065' was successfully enabled. 2023-02-13 12:22:20,512 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676290926065 in 474 msec 2023-02-13 12:22:20,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 12:22:20,794 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 92 completed 2023-02-13 12:22:20,794 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ReadOnlyZKClient(139): Connect 0x408a892a to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:20,807 DEBUG [Listener at localhost.localdomain/46179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59dfdeca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:20,810 DEBUG [hconnection-0x3082a08a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:20,813 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:40118, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:20,826 DEBUG [hconnection-0x3082a08a-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:20,832 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45284, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:20,921 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 12:22:20,927 DEBUG [hconnection-0x3082a08a-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:20,935 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:35542, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:21,064 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 12:22:21,225 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 12:22:21,356 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 12:22:21,467 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 12:22:22,876 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 12:22:23,957 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676290926065 2023-02-13 12:22:23,957 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676290926065 2023-02-13 12:22:23,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:23,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-13 12:22:23,962 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290943962"}]},"ts":"1676290943962"} 2023-02-13 12:22:23,964 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLING in hbase:meta 2023-02-13 12:22:24,019 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676290926065 to state=DISABLING 2023-02-13 12:22:24,020 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN}] 2023-02-13 12:22:24,024 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN 2023-02-13 12:22:24,025 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN 2023-02-13 12:22:24,025 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN 2023-02-13 12:22:24,025 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN 2023-02-13 12:22:24,026 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN 2023-02-13 12:22:24,026 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:24,026 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290944026"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290944026"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290944026"}]},"ts":"1676290944026"} 2023-02-13 12:22:24,028 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=106, state=RUNNABLE; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:24,029 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:24,029 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290944029"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290944029"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290944029"}]},"ts":"1676290944029"} 2023-02-13 12:22:24,030 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:24,030 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290944030"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290944030"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290944030"}]},"ts":"1676290944030"} 2023-02-13 12:22:24,031 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:24,031 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290944031"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290944031"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290944031"}]},"ts":"1676290944031"} 2023-02-13 12:22:24,031 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:24,031 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290944031"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290944031"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290944031"}]},"ts":"1676290944031"} 2023-02-13 12:22:24,036 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=111, state=RUNNABLE; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:24,037 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=110, state=RUNNABLE; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:24,038 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN 2023-02-13 12:22:24,040 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=109, state=RUNNABLE; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:24,040 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:24,040 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290944040"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290944040"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290944040"}]},"ts":"1676290944040"} 2023-02-13 12:22:24,041 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=108, state=RUNNABLE; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:24,045 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=107, state=RUNNABLE; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:24,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:24,183 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6e9099611b20e9378ce559b4b3a17188, disabling compactions & flushes 2023-02-13 12:22:24,183 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:24,183 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:24,183 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. after waiting 0 ms 2023-02-13 12:22:24,184 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:24,191 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:24,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 585751c2808ad67a24a61f19e9ac17f5, disabling compactions & flushes 2023-02-13 12:22:24,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:24,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:24,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. after waiting 0 ms 2023-02-13 12:22:24,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:24,195 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:24,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 197a58e90a6010ab47efab725a975e4c, disabling compactions & flushes 2023-02-13 12:22:24,195 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:24,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:24,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. after waiting 0 ms 2023-02-13 12:22:24,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:24,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-13 12:22:24,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 12:22:24,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:24,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 197a58e90a6010ab47efab725a975e4c: Waiting for close lock at 1676290944195Running coprocessor pre-close hooks at 1676290944195Disabling compacts and flushes for region at 1676290944195Disabling writes for close at 1676290944195Writing region close event to WAL at 1676290944219 (+24 ms)Running coprocessor post-close hooks at 1676290944223 (+4 ms)Closed at 1676290944224 (+1 ms) 2023-02-13 12:22:24,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:24,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:24,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 50cc7d0236be777dd672124c8ff93b56, disabling compactions & flushes 2023-02-13 12:22:24,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:24,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:24,227 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. after waiting 0 ms 2023-02-13 12:22:24,227 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:24,227 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSED 2023-02-13 12:22:24,227 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290944227"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290944227"}]},"ts":"1676290944227"} 2023-02-13 12:22:24,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 12:22:24,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 12:22:24,235 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:24,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 585751c2808ad67a24a61f19e9ac17f5: Waiting for close lock at 1676290944191Running coprocessor pre-close hooks at 1676290944191Disabling compacts and flushes for region at 1676290944191Disabling writes for close at 1676290944195 (+4 ms)Writing region close event to WAL at 1676290944212 (+17 ms)Running coprocessor post-close hooks at 1676290944233 (+21 ms)Closed at 1676290944235 (+2 ms) 2023-02-13 12:22:24,236 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:24,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6e9099611b20e9378ce559b4b3a17188: Waiting for close lock at 1676290944182Running coprocessor pre-close hooks at 1676290944182Disabling compacts and flushes for region at 1676290944182Disabling writes for close at 1676290944184 (+2 ms)Writing region close event to WAL at 1676290944219 (+35 ms)Running coprocessor post-close hooks at 1676290944235 (+16 ms)Closed at 1676290944236 (+1 ms) 2023-02-13 12:22:24,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:24,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:24,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81a0069ff66a08b05e0e5fe4210e0ce2, disabling compactions & flushes 2023-02-13 12:22:24,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:24,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:24,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. after waiting 0 ms 2023-02-13 12:22:24,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:24,239 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=110 2023-02-13 12:22:24,240 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=110, state=SUCCESS; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,36049,1676290923148 in 192 msec 2023-02-13 12:22:24,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:24,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:24,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6d6fd747f58897a134fa0a460eb9ec99, disabling compactions & flushes 2023-02-13 12:22:24,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:24,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:24,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. after waiting 0 ms 2023-02-13 12:22:24,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:24,241 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSED 2023-02-13 12:22:24,242 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290944241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290944241"}]},"ts":"1676290944241"} 2023-02-13 12:22:24,243 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSED 2023-02-13 12:22:24,243 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290944243"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290944243"}]},"ts":"1676290944243"} 2023-02-13 12:22:24,244 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN in 220 msec 2023-02-13 12:22:24,246 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=106 2023-02-13 12:22:24,247 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=106, state=SUCCESS; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,34139,1676290923061 in 216 msec 2023-02-13 12:22:24,248 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=111 2023-02-13 12:22:24,248 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN in 227 msec 2023-02-13 12:22:24,248 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=111, state=SUCCESS; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,34079,1676290923111 in 212 msec 2023-02-13 12:22:24,250 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN in 229 msec 2023-02-13 12:22:24,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 12:22:24,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:24,265 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 50cc7d0236be777dd672124c8ff93b56: Waiting for close lock at 1676290944226Running coprocessor pre-close hooks at 1676290944226Disabling compacts and flushes for region at 1676290944226Disabling writes for close at 1676290944227 (+1 ms)Writing region close event to WAL at 1676290944229 (+2 ms)Running coprocessor post-close hooks at 1676290944257 (+28 ms)Closed at 1676290944265 (+8 ms) 2023-02-13 12:22:24,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 12:22:24,268 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:24,268 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Waiting for close lock at 1676290944240Running coprocessor pre-close hooks at 1676290944240Disabling compacts and flushes for region at 1676290944240Disabling writes for close at 1676290944240Writing region close event to WAL at 1676290944256 (+16 ms)Running coprocessor post-close hooks at 1676290944267 (+11 ms)Closed at 1676290944268 (+1 ms) 2023-02-13 12:22:24,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:24,270 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSED 2023-02-13 12:22:24,271 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290944270"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290944270"}]},"ts":"1676290944270"} 2023-02-13 12:22:24,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 12:22:24,272 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:24,275 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:24,275 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6d6fd747f58897a134fa0a460eb9ec99: Waiting for close lock at 1676290944241Running coprocessor pre-close hooks at 1676290944241Disabling compacts and flushes for region at 1676290944241Disabling writes for close at 1676290944241Writing region close event to WAL at 1676290944265 (+24 ms)Running coprocessor post-close hooks at 1676290944273 (+8 ms)Closed at 1676290944275 (+2 ms) 2023-02-13 12:22:24,275 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSED 2023-02-13 12:22:24,275 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290944275"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290944275"}]},"ts":"1676290944275"} 2023-02-13 12:22:24,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:24,279 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSED 2023-02-13 12:22:24,279 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290944279"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290944279"}]},"ts":"1676290944279"} 2023-02-13 12:22:24,280 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=107 2023-02-13 12:22:24,280 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=107, state=SUCCESS; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,36049,1676290923148 in 231 msec 2023-02-13 12:22:24,281 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=109 2023-02-13 12:22:24,281 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=109, state=SUCCESS; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34139,1676290923061 in 238 msec 2023-02-13 12:22:24,284 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN in 260 msec 2023-02-13 12:22:24,288 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN in 261 msec 2023-02-13 12:22:24,289 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=108 2023-02-13 12:22:24,289 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=108, state=SUCCESS; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34079,1676290923111 in 240 msec 2023-02-13 12:22:24,295 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=108, resume processing ppid=105 2023-02-13 12:22:24,295 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN in 269 msec 2023-02-13 12:22:24,297 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290944296"}]},"ts":"1676290944296"} 2023-02-13 12:22:24,298 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLED in hbase:meta 2023-02-13 12:22:24,352 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676290926065 to state=DISABLED 2023-02-13 12:22:24,356 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676290926065 in 397 msec 2023-02-13 12:22:24,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-13 12:22:24,722 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 105 completed 2023-02-13 12:22:24,737 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676290926065-1676290944737 2023-02-13 12:22:24,738 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676290926065-1676290944737 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:24,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676290944738 (current time:1676290944738). 2023-02-13 12:22:24,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 12:22:24,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(139): Connect 0x74063279 to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:24,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40af1994, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:24,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:24,787 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:40124, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:24,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74063279 to 127.0.0.1:62541 2023-02-13 12:22:24,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:24,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 12:22:24,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676290926065-1676290944737 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:24,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:24,793 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:24,794 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE; timeout=600000, timestamp=1676291544794 2023-02-13 12:22:24,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676290926065-1676290944737 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:24,794 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676290926065-1676290944737 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676290926065 2023-02-13 12:22:24,795 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676290926065-1676290944737 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 12:22:24,795 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 12:22:24,833 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676290926065-1676290944737 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } 2023-02-13 12:22:24,836 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:24,837 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:24,837 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:24,837 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:24,837 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:24,837 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:24,837 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:24,837 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:24,837 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:24,837 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:24,838 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:24,838 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:24,838 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:24,838 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:24,839 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa] hfiles 2023-02-13 12:22:24,839 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa 2023-02-13 12:22:24,839 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827] hfiles 2023-02-13 12:22:24,839 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb] hfiles 2023-02-13 12:22:24,839 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827 2023-02-13 12:22:24,839 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb 2023-02-13 12:22:24,840 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8] hfiles 2023-02-13 12:22:24,840 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8 2023-02-13 12:22:24,842 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90] hfiles 2023-02-13 12:22:24,842 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585] hfiles 2023-02-13 12:22:24,842 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90 2023-02-13 12:22:24,842 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585 2023-02-13 12:22:24,847 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f] hfiles 2023-02-13 12:22:24,847 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a 2023-02-13 12:22:24,847 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 2023-02-13 12:22:24,847 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c 2023-02-13 12:22:24,847 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f 2023-02-13 12:22:24,848 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 2023-02-13 12:22:24,848 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f 2023-02-13 12:22:25,020 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676290926065-1676290944737 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } as finished. 2023-02-13 12:22:25,020 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676290926065-1676290944737 2023-02-13 12:22:25,021 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676290926065-1676290944737 2023-02-13 12:22:25,045 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:22:25,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676290926065-1676290944737 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } is done 2023-02-13 12:22:25,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-emptySnaptb-1676290926065-1676290944737 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 12:22:25,047 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 12:22:25,075 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676290926065-1676290944737 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676290926065-1676290944737 2023-02-13 12:22:25,102 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676290926065-1676290944737 of table testRestoreSnapshot-1676290926065 completed 2023-02-13 12:22:25,103 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676290926065-1676290944737 2023-02-13 12:22:25,103 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676290926065-1676290944737 2023-02-13 12:22:25,103 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676290926065-1676290944737 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676290926065 at 1676290944794Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676290926065-1676290944737 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } at 1676290944833 (+39 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676290926065-1676290944737 at 1676290945020 (+187 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676290926065-1676290944737 at 1676290945020Writing data manifest for hbase-failsafe-emptySnaptb-1676290926065-1676290944737 at 1676290945031 (+11 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676290926065-1676290944737 at 1676290945045 (+14 ms)Snapshot hbase-failsafe-emptySnaptb-1676290926065-1676290944737 of table testRestoreSnapshot-1676290926065 completed at 1676290945102 (+57 ms) 2023-02-13 12:22:25,106 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:25,107 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE in 315 msec 2023-02-13 12:22:25,547 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:22:25,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676290926065-1676290944737 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } is done 2023-02-13 12:22:25,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676290926065-1676290944737 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 12:22:25,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676290926065 snapshot=name: "emptySnaptb-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290928462 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 12:22:25,555 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676290926065 as table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:25,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-13 12:22:25,567 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/.tabledesc/.tableinfo.0000000003 2023-02-13 12:22:25,567 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/.tabledesc/.tableinfo.0000000002 2023-02-13 12:22:25,568 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/.tabledesc/.tableinfo.0000000003 2023-02-13 12:22:25,568 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } 2023-02-13 12:22:25,570 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290928462 type: DISABLED version: 2 ttl: 0 2023-02-13 12:22:25,570 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065 2023-02-13 12:22:25,577 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:25,581 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a 2023-02-13 12:22:25,582 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 2023-02-13 12:22:25,584 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c 2023-02-13 12:22:25,586 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f 2023-02-13 12:22:25,587 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 2023-02-13 12:22:25,588 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f 2023-02-13 12:22:25,589 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:25,589 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:25,589 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:25,589 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:25,589 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:25,589 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:25,595 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/3e370cf8b62343469a87e9049f031efb 2023-02-13 12:22:25,596 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/24b7ea7f7b1d42cfa460732691ead827 2023-02-13 12:22:25,596 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/4446486d315045788982bdebc8a2bb90 2023-02-13 12:22:25,597 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/31b72b71ccea46d8a373f1588b857585 2023-02-13 12:22:25,597 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/68333e5f91e3413fbabd03a7c9da49fa 2023-02-13 12:22:25,598 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/0a65255f37e3420c9fdfa034125fc5c8 2023-02-13 12:22:25,598 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290928462 type: DISABLED version: 2 ttl: 0 2023-02-13 12:22:25,599 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290945599"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:25,599 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290945599"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:25,600 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290945599"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:25,600 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290945599"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:25,600 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290945599"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:25,600 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290945599"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:25,602 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 12:22:25,602 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-13 12:22:25,603 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290945600"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290945600"}]},"ts":"1676290945600"} 2023-02-13 12:22:25,603 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290945600"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290945600"}]},"ts":"1676290945600"} 2023-02-13 12:22:25,603 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290945600"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290945600"}]},"ts":"1676290945600"} 2023-02-13 12:22:25,603 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290945600"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290945600"}]},"ts":"1676290945600"} 2023-02-13 12:22:25,603 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290945600"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290945600"}]},"ts":"1676290945600"} 2023-02-13 12:22:25,603 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290945600"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290945600"}]},"ts":"1676290945600"} 2023-02-13 12:22:25,605 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 12:22:25,605 INFO [PEWorker-1] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 12:22:25,605 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-13 12:22:25,605 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676290926065 completed! 2023-02-13 12:22:25,607 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676290926065 snapshot=name: "emptySnaptb-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290928462 type: DISABLED version: 2 ttl: 0 ) in 52 msec 2023-02-13 12:22:25,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-13 12:22:25,810 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676290926065, procId: 119 completed 2023-02-13 12:22:25,810 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676290926065-1676290944737 2023-02-13 12:22:25,811 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "hbase-failsafe-emptySnaptb-1676290926065-1676290944737" 2023-02-13 12:22:25,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676290926065-1676290944737 2023-02-13 12:22:25,815 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676290926065 2023-02-13 12:22:25,815 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testRestoreSnapshot-1676290926065 2023-02-13 12:22:25,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:25,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 12:22:25,820 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676290926065 2023-02-13 12:22:25,820 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290945820"}]},"ts":"1676290945820"} 2023-02-13 12:22:25,822 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLING in hbase:meta 2023-02-13 12:22:25,861 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 12:22:25,861 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN}] 2023-02-13 12:22:25,863 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN 2023-02-13 12:22:25,864 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN 2023-02-13 12:22:25,864 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN 2023-02-13 12:22:25,864 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN 2023-02-13 12:22:25,865 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:25,865 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:25,865 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN 2023-02-13 12:22:25,865 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:25,865 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:25,866 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:25,866 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN 2023-02-13 12:22:25,867 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:26,015 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 12:22:26,016 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 12:22:26,016 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 12:22:26,016 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 12:22:26,016 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 12:22:26,024 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:26,024 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:26,024 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:26,024 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290946024"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290946024"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290946024"}]},"ts":"1676290946024"} 2023-02-13 12:22:26,024 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290946024"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290946024"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290946024"}]},"ts":"1676290946024"} 2023-02-13 12:22:26,024 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:26,024 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:26,024 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290946024"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290946024"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290946024"}]},"ts":"1676290946024"} 2023-02-13 12:22:26,024 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290946023"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290946023"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290946023"}]},"ts":"1676290946023"} 2023-02-13 12:22:26,025 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290946024"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290946024"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290946024"}]},"ts":"1676290946024"} 2023-02-13 12:22:26,026 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=125, state=RUNNABLE; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:26,028 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=123, state=RUNNABLE; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:26,029 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=126, state=RUNNABLE; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:26,032 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:26,032 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=121, state=RUNNABLE; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:26,032 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290946032"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290946032"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290946032"}]},"ts":"1676290946032"} 2023-02-13 12:22:26,033 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=124, state=RUNNABLE; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:26,034 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=122, state=RUNNABLE; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:26,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 12:22:26,183 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:26,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:22:26,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:26,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:26,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:26,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:26,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:26,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:22:26,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:26,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:26,186 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:26,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:26,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:26,187 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:26,187 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:26,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:22:26,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:26,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:26,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:26,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:26,188 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:26,188 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:26,189 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 197a58e90a6010ab47efab725a975e4c columnFamilyName cf 2023-02-13 12:22:26,190 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(310): Store=197a58e90a6010ab47efab725a975e4c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:26,191 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:26,191 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:26,192 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 585751c2808ad67a24a61f19e9ac17f5 columnFamilyName cf 2023-02-13 12:22:26,192 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:26,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:26,193 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(310): Store=585751c2808ad67a24a61f19e9ac17f5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:26,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:26,194 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:26,194 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:26,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:26,194 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6e9099611b20e9378ce559b4b3a17188 columnFamilyName cf 2023-02-13 12:22:26,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:26,197 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:26,197 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 197a58e90a6010ab47efab725a975e4c; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58847543, jitterRate=-0.12310327589511871}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:26,197 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 197a58e90a6010ab47efab725a975e4c: Running coprocessor pre-open hook at 1676290946183Writing region info on filesystem at 1676290946183Initializing all the Stores at 1676290946185 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290946185Cleaning up temporary data from old regions at 1676290946194 (+9 ms)Cleaning up detritus from prior splits at 1676290946194Running coprocessor post-open hooks at 1676290946197 (+3 ms)Region opened successfully at 1676290946197 2023-02-13 12:22:26,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:26,199 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 585751c2808ad67a24a61f19e9ac17f5; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59461006, jitterRate=-0.11396196484565735}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:26,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 585751c2808ad67a24a61f19e9ac17f5: Running coprocessor pre-open hook at 1676290946186Writing region info on filesystem at 1676290946186Initializing all the Stores at 1676290946187 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290946187Cleaning up temporary data from old regions at 1676290946197 (+10 ms)Cleaning up detritus from prior splits at 1676290946197Running coprocessor post-open hooks at 1676290946199 (+2 ms)Region opened successfully at 1676290946199 2023-02-13 12:22:26,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c., pid=127, masterSystemTime=1676290946179 2023-02-13 12:22:26,200 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(310): Store=6e9099611b20e9378ce559b4b3a17188/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:26,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:26,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:26,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5., pid=129, masterSystemTime=1676290946182 2023-02-13 12:22:26,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:26,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:26,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:22:26,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:26,205 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:26,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:26,205 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290946205"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290946205"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290946205"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290946205"}]},"ts":"1676290946205"} 2023-02-13 12:22:26,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:26,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:26,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:26,208 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:26,208 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290946208"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290946208"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290946208"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290946208"}]},"ts":"1676290946208"} 2023-02-13 12:22:26,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:26,209 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:26,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:26,209 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:26,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:22:26,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:26,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:26,210 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6e9099611b20e9378ce559b4b3a17188; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61750294, jitterRate=-0.07984891533851624}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:26,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:26,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6e9099611b20e9378ce559b4b3a17188: Running coprocessor pre-open hook at 1676290946188Writing region info on filesystem at 1676290946188Initializing all the Stores at 1676290946190 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290946190Cleaning up temporary data from old regions at 1676290946206 (+16 ms)Cleaning up detritus from prior splits at 1676290946206Running coprocessor post-open hooks at 1676290946210 (+4 ms)Region opened successfully at 1676290946210 2023-02-13 12:22:26,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:26,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188., pid=130, masterSystemTime=1676290946185 2023-02-13 12:22:26,211 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=125 2023-02-13 12:22:26,211 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=125, state=SUCCESS; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111 in 182 msec 2023-02-13 12:22:26,212 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:26,214 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:26,215 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:26,216 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:26,216 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:26,215 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:26,217 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:26,216 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 50cc7d0236be777dd672124c8ff93b56 columnFamilyName cf 2023-02-13 12:22:26,216 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:26,216 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN in 351 msec 2023-02-13 12:22:26,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:26,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:22:26,217 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d6fd747f58897a134fa0a460eb9ec99 columnFamilyName cf 2023-02-13 12:22:26,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:26,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:26,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:26,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:26,218 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(310): Store=6d6fd747f58897a134fa0a460eb9ec99/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:26,219 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=126 2023-02-13 12:22:26,219 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=126, state=SUCCESS; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,34139,1676290923061 in 181 msec 2023-02-13 12:22:26,219 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(310): Store=50cc7d0236be777dd672124c8ff93b56/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:26,220 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:26,220 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290946220"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290946220"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290946220"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290946220"}]},"ts":"1676290946220"} 2023-02-13 12:22:26,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:26,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:26,220 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:26,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:26,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:26,223 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:26,223 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:26,223 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN in 358 msec 2023-02-13 12:22:26,224 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81a0069ff66a08b05e0e5fe4210e0ce2 columnFamilyName cf 2023-02-13 12:22:26,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:26,225 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(310): Store=81a0069ff66a08b05e0e5fe4210e0ce2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:26,226 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=121 2023-02-13 12:22:26,226 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=121, state=SUCCESS; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148 in 191 msec 2023-02-13 12:22:26,226 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 50cc7d0236be777dd672124c8ff93b56; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65693660, jitterRate=-0.02108818292617798}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:26,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:26,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 50cc7d0236be777dd672124c8ff93b56: Running coprocessor pre-open hook at 1676290946206Writing region info on filesystem at 1676290946206Initializing all the Stores at 1676290946207 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290946207Cleaning up temporary data from old regions at 1676290946222 (+15 ms)Cleaning up detritus from prior splits at 1676290946222Running coprocessor post-open hooks at 1676290946226 (+4 ms)Region opened successfully at 1676290946226 2023-02-13 12:22:26,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:26,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56., pid=132, masterSystemTime=1676290946179 2023-02-13 12:22:26,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6d6fd747f58897a134fa0a460eb9ec99; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64010272, jitterRate=-0.0461726188659668}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:26,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6d6fd747f58897a134fa0a460eb9ec99: Running coprocessor pre-open hook at 1676290946211Writing region info on filesystem at 1676290946211Initializing all the Stores at 1676290946211Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290946211Cleaning up temporary data from old regions at 1676290946223 (+12 ms)Cleaning up detritus from prior splits at 1676290946224 (+1 ms)Running coprocessor post-open hooks at 1676290946227 (+3 ms)Region opened successfully at 1676290946227 2023-02-13 12:22:26,227 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN in 365 msec 2023-02-13 12:22:26,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99., pid=128, masterSystemTime=1676290946182 2023-02-13 12:22:26,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:26,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:26,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:26,230 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:26,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:26,231 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290946230"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290946230"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290946230"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290946230"}]},"ts":"1676290946230"} 2023-02-13 12:22:26,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:26,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:26,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81a0069ff66a08b05e0e5fe4210e0ce2; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68817964, jitterRate=0.02546757459640503}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:26,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Running coprocessor pre-open hook at 1676290946218Writing region info on filesystem at 1676290946218Initializing all the Stores at 1676290946220 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290946220Cleaning up temporary data from old regions at 1676290946228 (+8 ms)Cleaning up detritus from prior splits at 1676290946229 (+1 ms)Running coprocessor post-open hooks at 1676290946232 (+3 ms)Region opened successfully at 1676290946232 2023-02-13 12:22:26,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2., pid=131, masterSystemTime=1676290946185 2023-02-13 12:22:26,238 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:26,238 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290946238"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290946238"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290946238"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290946238"}]},"ts":"1676290946238"} 2023-02-13 12:22:26,241 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:26,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:26,241 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290946241"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290946241"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290946241"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290946241"}]},"ts":"1676290946241"} 2023-02-13 12:22:26,241 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:26,242 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=122 2023-02-13 12:22:26,242 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=122, state=SUCCESS; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34079,1676290923111 in 202 msec 2023-02-13 12:22:26,246 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=123 2023-02-13 12:22:26,247 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=123, state=SUCCESS; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34139,1676290923061 in 213 msec 2023-02-13 12:22:26,247 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN in 381 msec 2023-02-13 12:22:26,253 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=124 2023-02-13 12:22:26,254 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=124, state=SUCCESS; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,36049,1676290923148 in 211 msec 2023-02-13 12:22:26,254 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN in 386 msec 2023-02-13 12:22:26,257 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=124, resume processing ppid=120 2023-02-13 12:22:26,257 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN in 393 msec 2023-02-13 12:22:26,258 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290946258"}]},"ts":"1676290946258"} 2023-02-13 12:22:26,260 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLED in hbase:meta 2023-02-13 12:22:26,273 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676290926065' was successfully enabled. 2023-02-13 12:22:26,276 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676290926065 in 459 msec 2023-02-13 12:22:26,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 12:22:26,572 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 120 completed 2023-02-13 12:22:26,572 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ReadOnlyZKClient(139): Connect 0x1b628e69 to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:26,586 DEBUG [Listener at localhost.localdomain/46179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bef6796, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:26,589 DEBUG [hconnection-0x7320d9d2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:26,599 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:40130, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:26,664 DEBUG [hconnection-0x7320d9d2-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:26,671 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:35546, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:26,672 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 12:22:26,701 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 12:22:26,726 DEBUG [hconnection-0x7320d9d2-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:26,750 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45294, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:26,753 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 12:22:26,785 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 12:22:26,812 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 12:22:26,819 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676290926065 2023-02-13 12:22:26,820 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676290926065 2023-02-13 12:22:26,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:26,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-13 12:22:26,827 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290946827"}]},"ts":"1676290946827"} 2023-02-13 12:22:26,829 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLING in hbase:meta 2023-02-13 12:22:26,840 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676290926065 to state=DISABLING 2023-02-13 12:22:26,841 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN}] 2023-02-13 12:22:26,844 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN 2023-02-13 12:22:26,845 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN 2023-02-13 12:22:26,845 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN 2023-02-13 12:22:26,845 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN 2023-02-13 12:22:26,845 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN 2023-02-13 12:22:26,846 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:26,846 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:26,847 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290946846"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290946846"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290946846"}]},"ts":"1676290946846"} 2023-02-13 12:22:26,847 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:26,847 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290946847"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290946847"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290946847"}]},"ts":"1676290946847"} 2023-02-13 12:22:26,847 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:26,847 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290946847"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290946847"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290946847"}]},"ts":"1676290946847"} 2023-02-13 12:22:26,848 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:26,848 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290946848"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290946848"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290946848"}]},"ts":"1676290946848"} 2023-02-13 12:22:26,849 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=139, state=RUNNABLE; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:26,852 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=134, state=RUNNABLE; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:26,854 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=137, state=RUNNABLE; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:26,854 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=138, state=RUNNABLE; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:26,847 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290946846"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290946846"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290946846"}]},"ts":"1676290946846"} 2023-02-13 12:22:26,857 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN 2023-02-13 12:22:26,858 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=135, state=RUNNABLE; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:26,858 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:26,858 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290946858"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290946858"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290946858"}]},"ts":"1676290946858"} 2023-02-13 12:22:26,859 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=136, state=RUNNABLE; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:27,004 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:27,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 585751c2808ad67a24a61f19e9ac17f5, disabling compactions & flushes 2023-02-13 12:22:27,005 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:27,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:27,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. after waiting 0 ms 2023-02-13 12:22:27,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:27,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:27,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6e9099611b20e9378ce559b4b3a17188, disabling compactions & flushes 2023-02-13 12:22:27,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:27,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:27,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. after waiting 0 ms 2023-02-13 12:22:27,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:27,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:27,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 50cc7d0236be777dd672124c8ff93b56, disabling compactions & flushes 2023-02-13 12:22:27,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:27,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:27,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. after waiting 0 ms 2023-02-13 12:22:27,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:27,024 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 12:22:27,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 12:22:27,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:27,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 585751c2808ad67a24a61f19e9ac17f5: Waiting for close lock at 1676290947004Running coprocessor pre-close hooks at 1676290947004Disabling compacts and flushes for region at 1676290947004Disabling writes for close at 1676290947005 (+1 ms)Writing region close event to WAL at 1676290947005Running coprocessor post-close hooks at 1676290947026 (+21 ms)Closed at 1676290947026 2023-02-13 12:22:27,027 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:27,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6e9099611b20e9378ce559b4b3a17188: Waiting for close lock at 1676290947006Running coprocessor pre-close hooks at 1676290947006Disabling compacts and flushes for region at 1676290947006Disabling writes for close at 1676290947007 (+1 ms)Writing region close event to WAL at 1676290947012 (+5 ms)Running coprocessor post-close hooks at 1676290947027 (+15 ms)Closed at 1676290947027 2023-02-13 12:22:27,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:27,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:27,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6d6fd747f58897a134fa0a460eb9ec99, disabling compactions & flushes 2023-02-13 12:22:27,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:27,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:27,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. after waiting 0 ms 2023-02-13 12:22:27,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:27,033 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSED 2023-02-13 12:22:27,033 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290947033"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290947033"}]},"ts":"1676290947033"} 2023-02-13 12:22:27,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:27,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:27,035 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSED 2023-02-13 12:22:27,035 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290947035"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290947035"}]},"ts":"1676290947035"} 2023-02-13 12:22:27,039 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=139 2023-02-13 12:22:27,039 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=139, state=SUCCESS; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,34139,1676290923061 in 186 msec 2023-02-13 12:22:27,042 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=134 2023-02-13 12:22:27,043 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN in 198 msec 2023-02-13 12:22:27,043 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=134, state=SUCCESS; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,36049,1676290923148 in 187 msec 2023-02-13 12:22:27,050 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN in 202 msec 2023-02-13 12:22:27,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81a0069ff66a08b05e0e5fe4210e0ce2, disabling compactions & flushes 2023-02-13 12:22:27,061 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:27,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:27,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. after waiting 0 ms 2023-02-13 12:22:27,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:27,062 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 12:22:27,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 12:22:27,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:27,064 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 50cc7d0236be777dd672124c8ff93b56: Waiting for close lock at 1676290947008Running coprocessor pre-close hooks at 1676290947008Disabling compacts and flushes for region at 1676290947008Disabling writes for close at 1676290947009 (+1 ms)Writing region close event to WAL at 1676290947024 (+15 ms)Running coprocessor post-close hooks at 1676290947063 (+39 ms)Closed at 1676290947063 2023-02-13 12:22:27,064 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:27,064 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6d6fd747f58897a134fa0a460eb9ec99: Waiting for close lock at 1676290947029Running coprocessor pre-close hooks at 1676290947029Disabling compacts and flushes for region at 1676290947029Disabling writes for close at 1676290947030 (+1 ms)Writing region close event to WAL at 1676290947033 (+3 ms)Running coprocessor post-close hooks at 1676290947064 (+31 ms)Closed at 1676290947064 2023-02-13 12:22:27,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:27,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:27,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 197a58e90a6010ab47efab725a975e4c, disabling compactions & flushes 2023-02-13 12:22:27,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:27,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:27,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. after waiting 0 ms 2023-02-13 12:22:27,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:27,073 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSED 2023-02-13 12:22:27,074 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290947073"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290947073"}]},"ts":"1676290947073"} 2023-02-13 12:22:27,075 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:27,076 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSED 2023-02-13 12:22:27,076 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290947076"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290947076"}]},"ts":"1676290947076"} 2023-02-13 12:22:27,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-13 12:22:27,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 12:22:27,084 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=135 2023-02-13 12:22:27,084 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=135, state=SUCCESS; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34079,1676290923111 in 219 msec 2023-02-13 12:22:27,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:27,085 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Waiting for close lock at 1676290947034Running coprocessor pre-close hooks at 1676290947034Disabling compacts and flushes for region at 1676290947034Disabling writes for close at 1676290947061 (+27 ms)Writing region close event to WAL at 1676290947065 (+4 ms)Running coprocessor post-close hooks at 1676290947085 (+20 ms)Closed at 1676290947085 2023-02-13 12:22:27,087 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=136 2023-02-13 12:22:27,087 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=136, state=SUCCESS; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,34139,1676290923061 in 219 msec 2023-02-13 12:22:27,087 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 12:22:27,087 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:27,088 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:27,088 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 197a58e90a6010ab47efab725a975e4c: Waiting for close lock at 1676290947068Running coprocessor pre-close hooks at 1676290947068Disabling compacts and flushes for region at 1676290947068Disabling writes for close at 1676290947069 (+1 ms)Writing region close event to WAL at 1676290947080 (+11 ms)Running coprocessor post-close hooks at 1676290947088 (+8 ms)Closed at 1676290947088 2023-02-13 12:22:27,088 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSED 2023-02-13 12:22:27,088 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN in 244 msec 2023-02-13 12:22:27,088 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290947088"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290947088"}]},"ts":"1676290947088"} 2023-02-13 12:22:27,090 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN in 246 msec 2023-02-13 12:22:27,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:27,091 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSED 2023-02-13 12:22:27,092 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290947091"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290947091"}]},"ts":"1676290947091"} 2023-02-13 12:22:27,094 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=137 2023-02-13 12:22:27,095 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=137, state=SUCCESS; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,36049,1676290923148 in 237 msec 2023-02-13 12:22:27,096 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN in 254 msec 2023-02-13 12:22:27,097 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=138 2023-02-13 12:22:27,097 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=138, state=SUCCESS; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34079,1676290923111 in 241 msec 2023-02-13 12:22:27,100 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=138, resume processing ppid=133 2023-02-13 12:22:27,100 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN in 256 msec 2023-02-13 12:22:27,102 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290947102"}]},"ts":"1676290947102"} 2023-02-13 12:22:27,105 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLED in hbase:meta 2023-02-13 12:22:27,115 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676290926065 to state=DISABLED 2023-02-13 12:22:27,118 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676290926065 in 296 msec 2023-02-13 12:22:27,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-13 12:22:27,580 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 133 completed 2023-02-13 12:22:27,594 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676290926065-1676290947594 2023-02-13 12:22:27,596 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676290926065-1676290947594 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:27,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676290947596 (current time:1676290947596). 2023-02-13 12:22:27,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 12:22:27,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(139): Connect 0x06b549f8 to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:27,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34a50599, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:27,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:27,639 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:40144, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:27,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06b549f8 to 127.0.0.1:62541 2023-02-13 12:22:27,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:27,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 12:22:27,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676290926065-1676290947594 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:27,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:27,643 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:27,645 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE; timeout=600000, timestamp=1676291547645 2023-02-13 12:22:27,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676290926065-1676290947594 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } 2023-02-13 12:22:27,645 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676290926065-1676290947594 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676290926065 2023-02-13 12:22:27,645 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676290926065-1676290947594 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 12:22:27,645 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 12:22:27,668 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676290926065-1676290947594 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } 2023-02-13 12:22:27,677 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:27,677 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:27,678 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:27,678 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:27,680 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:27,680 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:27,680 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:27,681 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:27,681 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:27,681 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:27,682 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:27,682 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:27,682 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:27,682 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:27,682 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:27,683 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:27,683 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:27,684 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:27,687 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:27,693 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:27,697 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:27,846 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676290926065-1676290947594 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } as finished. 2023-02-13 12:22:27,846 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676290926065-1676290947594 2023-02-13 12:22:27,847 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676290926065-1676290947594 2023-02-13 12:22:27,895 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:22:27,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676290926065-1676290947594 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } is done 2023-02-13 12:22:27,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snaptb1-1676290926065-1676290947594 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 12:22:27,898 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 12:22:27,936 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676290926065-1676290947594 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/hbase-failsafe-snaptb1-1676290926065-1676290947594 2023-02-13 12:22:27,972 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676290926065-1676290947594 of table testRestoreSnapshot-1676290926065 completed 2023-02-13 12:22:27,972 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676290926065-1676290947594 2023-02-13 12:22:27,973 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676290926065-1676290947594 2023-02-13 12:22:27,973 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676290926065-1676290947594 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676290926065 at 1676290947645Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676290926065-1676290947594 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } at 1676290947668 (+23 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676290926065-1676290947594 at 1676290947846 (+178 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676290926065-1676290947594 at 1676290947846Writing data manifest for hbase-failsafe-snaptb1-1676290926065-1676290947594 at 1676290947861 (+15 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676290926065-1676290947594 at 1676290947895 (+34 ms)Snapshot hbase-failsafe-snaptb1-1676290926065-1676290947594 of table testRestoreSnapshot-1676290926065 completed at 1676290947972 (+77 ms) 2023-02-13 12:22:27,975 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE 2023-02-13 12:22:27,978 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676290926065, type=EXCLUSIVE in 334 msec 2023-02-13 12:22:28,398 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:22:28,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676290926065-1676290947594 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 } is done 2023-02-13 12:22:28,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676290926065-1676290947594 table=testRestoreSnapshot-1676290926065 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 12:22:28,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676290926065 snapshot=name: "snaptb1-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290938618 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 12:22:28,421 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676290926065 as table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-13 12:22:28,433 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/.tabledesc/.tableinfo.0000000004 2023-02-13 12:22:28,433 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/.tabledesc/.tableinfo.0000000003 2023-02-13 12:22:28,434 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/.tabledesc/.tableinfo.0000000004 2023-02-13 12:22:28,434 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } 2023-02-13 12:22:28,437 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290938618 type: DISABLED version: 2 ttl: 0 2023-02-13 12:22:28,438 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,445 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,447 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a of snapshot=snaptb1-1676290926065 to region=c29be10d60d1b4c8e95e9f2610cc0081 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,450 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c of snapshot=snaptb1-1676290926065 to region=c29be10d60d1b4c8e95e9f2610cc0081 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,454 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 of snapshot=snaptb1-1676290926065 to region=c29be10d60d1b4c8e95e9f2610cc0081 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,456 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 of snapshot=snaptb1-1676290926065 to region=c29be10d60d1b4c8e95e9f2610cc0081 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,459 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c of snapshot=snaptb1-1676290926065 to region=c29be10d60d1b4c8e95e9f2610cc0081 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,463 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af of snapshot=snaptb1-1676290926065 to region=c29be10d60d1b4c8e95e9f2610cc0081 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,466 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e of snapshot=snaptb1-1676290926065 to region=c29be10d60d1b4c8e95e9f2610cc0081 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,469 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f of snapshot=snaptb1-1676290926065 to region=c29be10d60d1b4c8e95e9f2610cc0081 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,472 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 of snapshot=snaptb1-1676290926065 to region=c29be10d60d1b4c8e95e9f2610cc0081 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,475 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 of snapshot=snaptb1-1676290926065 to region=c29be10d60d1b4c8e95e9f2610cc0081 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,478 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 of snapshot=snaptb1-1676290926065 to region=c29be10d60d1b4c8e95e9f2610cc0081 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,480 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f of snapshot=snaptb1-1676290926065 to region=c29be10d60d1b4c8e95e9f2610cc0081 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,483 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:28,483 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:28,483 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:28,483 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:28,483 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:28,483 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:28,488 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 0a65255f37e3420c9fdfa034125fc5c8 of snapshot=snaptb1-1676290926065 to region=6e9099611b20e9378ce559b4b3a17188 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,490 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 05ca7eda38a34d9c85aaeaa016b874da of snapshot=snaptb1-1676290926065 to region=50cc7d0236be777dd672124c8ff93b56 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,490 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 4446486d315045788982bdebc8a2bb90 of snapshot=snaptb1-1676290926065 to region=6d6fd747f58897a134fa0a460eb9ec99 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,490 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 189c775dee9b461ca79ff1ad57032a6e of snapshot=snaptb1-1676290926065 to region=197a58e90a6010ab47efab725a975e4c table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,490 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 68333e5f91e3413fbabd03a7c9da49fa of snapshot=snaptb1-1676290926065 to region=585751c2808ad67a24a61f19e9ac17f5 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,490 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 31b72b71ccea46d8a373f1588b857585 of snapshot=snaptb1-1676290926065 to region=81a0069ff66a08b05e0e5fe4210e0ce2 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,500 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 79d6c194793c4877a70ea1a6905cd846 of snapshot=snaptb1-1676290926065 to region=585751c2808ad67a24a61f19e9ac17f5 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,500 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d1029a30e4964e55bcbe8195fe1e7315 of snapshot=snaptb1-1676290926065 to region=6d6fd747f58897a134fa0a460eb9ec99 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,500 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 84c9c4a754f148fc8187a50ff196c186 of snapshot=snaptb1-1676290926065 to region=81a0069ff66a08b05e0e5fe4210e0ce2 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,504 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 3e370cf8b62343469a87e9049f031efb of snapshot=snaptb1-1676290926065 to region=50cc7d0236be777dd672124c8ff93b56 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,504 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 48186611db074ebeab6e0941597e1c9e of snapshot=snaptb1-1676290926065 to region=6e9099611b20e9378ce559b4b3a17188 table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,504 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 24b7ea7f7b1d42cfa460732691ead827 of snapshot=snaptb1-1676290926065 to region=197a58e90a6010ab47efab725a975e4c table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,514 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290938618 type: DISABLED version: 2 ttl: 0 2023-02-13 12:22:28,515 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290948515"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:28,515 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290948515"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:28,515 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290948515"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:28,515 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290948515"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:28,515 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290948515"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:28,515 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290948515"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:28,517 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 12:22:28,517 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-13 12:22:28,517 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290948516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290948516"}]},"ts":"1676290948516"} 2023-02-13 12:22:28,518 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290948516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290948516"}]},"ts":"1676290948516"} 2023-02-13 12:22:28,518 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290948516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290948516"}]},"ts":"1676290948516"} 2023-02-13 12:22:28,518 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290948516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290948516"}]},"ts":"1676290948516"} 2023-02-13 12:22:28,518 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290948516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290948516"}]},"ts":"1676290948516"} 2023-02-13 12:22:28,518 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290948516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290948516"}]},"ts":"1676290948516"} 2023-02-13 12:22:28,520 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 12:22:28,520 INFO [PEWorker-1] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 12:22:28,520 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'}] 2023-02-13 12:22:28,520 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676290926065 table=testRestoreSnapshot-1676290926065 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676290926065 completed! 2023-02-13 12:22:28,522 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676290926065 snapshot=name: "snaptb1-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290938618 type: DISABLED version: 2 ttl: 0 ) in 103 msec 2023-02-13 12:22:28,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-13 12:22:28,674 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676290926065, procId: 147 completed 2023-02-13 12:22:28,674 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676290926065-1676290947594 2023-02-13 12:22:28,675 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "hbase-failsafe-snaptb1-1676290926065-1676290947594" 2023-02-13 12:22:28,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676290926065-1676290947594 2023-02-13 12:22:28,680 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,680 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 12:22:28,685 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676290926065 2023-02-13 12:22:28,685 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290948685"}]},"ts":"1676290948685"} 2023-02-13 12:22:28,687 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLING in hbase:meta 2023-02-13 12:22:28,720 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 12:22:28,720 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN}] 2023-02-13 12:22:28,722 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN 2023-02-13 12:22:28,722 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN 2023-02-13 12:22:28,722 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN 2023-02-13 12:22:28,723 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN 2023-02-13 12:22:28,723 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN 2023-02-13 12:22:28,723 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:28,723 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:28,723 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:28,723 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:28,724 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:28,724 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN 2023-02-13 12:22:28,725 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 12:22:28,874 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 12:22:28,874 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 12:22:28,874 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 12:22:28,874 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 12:22:28,874 DEBUG [jenkins-hbase16:46677] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 12:22:28,879 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:28,879 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:28,880 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290948879"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290948879"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290948879"}]},"ts":"1676290948879"} 2023-02-13 12:22:28,880 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:28,880 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290948880"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290948880"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290948880"}]},"ts":"1676290948880"} 2023-02-13 12:22:28,880 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:28,880 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290948880"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290948880"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290948880"}]},"ts":"1676290948880"} 2023-02-13 12:22:28,879 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:28,880 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290948879"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290948879"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290948879"}]},"ts":"1676290948879"} 2023-02-13 12:22:28,880 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290948879"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290948879"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290948879"}]},"ts":"1676290948879"} 2023-02-13 12:22:28,881 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=154, state=RUNNABLE; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:28,884 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=151, state=RUNNABLE; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:28,886 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=152, state=RUNNABLE; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:28,889 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:28,889 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290948889"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290948889"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290948889"}]},"ts":"1676290948889"} 2023-02-13 12:22:28,893 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=153, state=RUNNABLE; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:28,894 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=150, state=RUNNABLE; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:28,896 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=149, state=RUNNABLE; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:28,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 12:22:29,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:29,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:22:29,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:29,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:29,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:29,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:29,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:29,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:22:29,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:29,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:29,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:29,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:29,047 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:29,047 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:29,048 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:29,048 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:29,048 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:29,048 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:29,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:22:29,048 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81a0069ff66a08b05e0e5fe4210e0ce2 columnFamilyName cf 2023-02-13 12:22:29,048 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:29,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:29,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:29,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:29,049 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 585751c2808ad67a24a61f19e9ac17f5 columnFamilyName cf 2023-02-13 12:22:29,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:29,051 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:29,054 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:29,054 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:29,054 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 50cc7d0236be777dd672124c8ff93b56 columnFamilyName cf 2023-02-13 12:22:29,096 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-05ca7eda38a34d9c85aaeaa016b874da 2023-02-13 12:22:29,096 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-31b72b71ccea46d8a373f1588b857585 2023-02-13 12:22:29,102 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-68333e5f91e3413fbabd03a7c9da49fa 2023-02-13 12:22:29,114 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-84c9c4a754f148fc8187a50ff196c186 2023-02-13 12:22:29,114 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-3e370cf8b62343469a87e9049f031efb 2023-02-13 12:22:29,114 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(310): Store=81a0069ff66a08b05e0e5fe4210e0ce2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:29,114 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(310): Store=50cc7d0236be777dd672124c8ff93b56/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:29,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:29,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:29,117 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-79d6c194793c4877a70ea1a6905cd846 2023-02-13 12:22:29,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:29,117 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(310): Store=585751c2808ad67a24a61f19e9ac17f5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:29,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:29,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:29,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:29,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:29,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:29,120 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 50cc7d0236be777dd672124c8ff93b56; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72377865, jitterRate=0.0785142332315445}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:29,120 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81a0069ff66a08b05e0e5fe4210e0ce2; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69428635, jitterRate=0.03456728160381317}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:29,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 50cc7d0236be777dd672124c8ff93b56: Running coprocessor pre-open hook at 1676290949049Writing region info on filesystem at 1676290949049Initializing all the Stores at 1676290949049Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290949050 (+1 ms)Cleaning up temporary data from old regions at 1676290949117 (+67 ms)Cleaning up detritus from prior splits at 1676290949118 (+1 ms)Running coprocessor post-open hooks at 1676290949120 (+2 ms)Region opened successfully at 1676290949120 2023-02-13 12:22:29,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Running coprocessor pre-open hook at 1676290949045Writing region info on filesystem at 1676290949045Initializing all the Stores at 1676290949045Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290949046 (+1 ms)Cleaning up temporary data from old regions at 1676290949117 (+71 ms)Cleaning up detritus from prior splits at 1676290949118 (+1 ms)Running coprocessor post-open hooks at 1676290949120 (+2 ms)Region opened successfully at 1676290949120 2023-02-13 12:22:29,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56., pid=159, masterSystemTime=1676290949045 2023-02-13 12:22:29,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2., pid=157, masterSystemTime=1676290949040 2023-02-13 12:22:29,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:29,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:29,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:29,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:29,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:22:29,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:29,123 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:29,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:29,124 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:29,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:29,124 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290949123"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290949123"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290949123"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290949123"}]},"ts":"1676290949123"} 2023-02-13 12:22:29,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:29,124 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 585751c2808ad67a24a61f19e9ac17f5; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62730113, jitterRate=-0.06524847447872162}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:29,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:29,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 585751c2808ad67a24a61f19e9ac17f5: Running coprocessor pre-open hook at 1676290949039Writing region info on filesystem at 1676290949039Initializing all the Stores at 1676290949040 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290949040Cleaning up temporary data from old regions at 1676290949119 (+79 ms)Cleaning up detritus from prior splits at 1676290949119Running coprocessor post-open hooks at 1676290949124 (+5 ms)Region opened successfully at 1676290949124 2023-02-13 12:22:29,124 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:29,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:22:29,125 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:29,125 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290949124"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290949124"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290949124"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290949124"}]},"ts":"1676290949124"} 2023-02-13 12:22:29,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:29,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:29,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:29,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:29,126 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5., pid=155, masterSystemTime=1676290949035 2023-02-13 12:22:29,126 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:29,127 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:29,127 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:29,128 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:29,128 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:29,128 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:29,128 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:22:29,128 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:29,128 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:29,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:29,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:29,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:29,129 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:29,129 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290949129"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290949129"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290949129"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290949129"}]},"ts":"1676290949129"} 2023-02-13 12:22:29,129 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6e9099611b20e9378ce559b4b3a17188 columnFamilyName cf 2023-02-13 12:22:29,130 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:29,132 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:29,133 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:29,133 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=150 2023-02-13 12:22:29,133 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=150, state=SUCCESS; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061 in 234 msec 2023-02-13 12:22:29,132 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=152 2023-02-13 12:22:29,134 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=152, state=SUCCESS; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34079,1676290923111 in 241 msec 2023-02-13 12:22:29,134 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN in 413 msec 2023-02-13 12:22:29,134 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN in 412 msec 2023-02-13 12:22:29,134 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=154 2023-02-13 12:22:29,134 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=154, state=SUCCESS; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148 in 250 msec 2023-02-13 12:22:29,134 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 197a58e90a6010ab47efab725a975e4c columnFamilyName cf 2023-02-13 12:22:29,135 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:29,135 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:29,135 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN in 414 msec 2023-02-13 12:22:29,136 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d6fd747f58897a134fa0a460eb9ec99 columnFamilyName cf 2023-02-13 12:22:29,161 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-0a65255f37e3420c9fdfa034125fc5c8 2023-02-13 12:22:29,161 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-4446486d315045788982bdebc8a2bb90 2023-02-13 12:22:29,165 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-189c775dee9b461ca79ff1ad57032a6e 2023-02-13 12:22:29,178 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-48186611db074ebeab6e0941597e1c9e 2023-02-13 12:22:29,179 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-d1029a30e4964e55bcbe8195fe1e7315 2023-02-13 12:22:29,180 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(310): Store=6e9099611b20e9378ce559b4b3a17188/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:29,180 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(310): Store=6d6fd747f58897a134fa0a460eb9ec99/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:29,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:29,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:29,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:29,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:29,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:29,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:29,187 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6d6fd747f58897a134fa0a460eb9ec99; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69475221, jitterRate=0.03526146709918976}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:29,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6d6fd747f58897a134fa0a460eb9ec99: Running coprocessor pre-open hook at 1676290949129Writing region info on filesystem at 1676290949129Initializing all the Stores at 1676290949130 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290949131 (+1 ms)Cleaning up temporary data from old regions at 1676290949184 (+53 ms)Cleaning up detritus from prior splits at 1676290949185 (+1 ms)Running coprocessor post-open hooks at 1676290949187 (+2 ms)Region opened successfully at 1676290949187 2023-02-13 12:22:29,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6e9099611b20e9378ce559b4b3a17188; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65950309, jitterRate=-0.01726381480693817}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:29,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6e9099611b20e9378ce559b4b3a17188: Running coprocessor pre-open hook at 1676290949124Writing region info on filesystem at 1676290949124Initializing all the Stores at 1676290949125 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290949125Cleaning up temporary data from old regions at 1676290949184 (+59 ms)Cleaning up detritus from prior splits at 1676290949185 (+1 ms)Running coprocessor post-open hooks at 1676290949188 (+3 ms)Region opened successfully at 1676290949188 2023-02-13 12:22:29,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99., pid=156, masterSystemTime=1676290949035 2023-02-13 12:22:29,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188., pid=160, masterSystemTime=1676290949040 2023-02-13 12:22:29,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:29,191 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:29,191 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:29,191 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-24b7ea7f7b1d42cfa460732691ead827 2023-02-13 12:22:29,191 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290949191"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290949191"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290949191"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290949191"}]},"ts":"1676290949191"} 2023-02-13 12:22:29,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:29,191 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:29,192 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:29,192 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290949192"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290949192"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290949192"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290949192"}]},"ts":"1676290949192"} 2023-02-13 12:22:29,193 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(310): Store=197a58e90a6010ab47efab725a975e4c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:29,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:29,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:29,197 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=151 2023-02-13 12:22:29,197 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=151, state=SUCCESS; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,36049,1676290923148 in 310 msec 2023-02-13 12:22:29,199 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=149 2023-02-13 12:22:29,199 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=149, state=SUCCESS; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,34079,1676290923111 in 300 msec 2023-02-13 12:22:29,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:29,200 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN in 478 msec 2023-02-13 12:22:29,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 197a58e90a6010ab47efab725a975e4c; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74288834, jitterRate=0.10698989033699036}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:29,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 197a58e90a6010ab47efab725a975e4c: Running coprocessor pre-open hook at 1676290949125Writing region info on filesystem at 1676290949125Initializing all the Stores at 1676290949126 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290949126Cleaning up temporary data from old regions at 1676290949196 (+70 ms)Cleaning up detritus from prior splits at 1676290949197 (+1 ms)Running coprocessor post-open hooks at 1676290949200 (+3 ms)Region opened successfully at 1676290949200 2023-02-13 12:22:29,201 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN in 479 msec 2023-02-13 12:22:29,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c., pid=158, masterSystemTime=1676290949045 2023-02-13 12:22:29,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:29,202 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:29,203 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:29,203 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290949203"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290949203"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290949203"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290949203"}]},"ts":"1676290949203"} 2023-02-13 12:22:29,206 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=153 2023-02-13 12:22:29,206 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=153, state=SUCCESS; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34139,1676290923061 in 312 msec 2023-02-13 12:22:29,207 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=153, resume processing ppid=148 2023-02-13 12:22:29,207 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN in 486 msec 2023-02-13 12:22:29,208 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290949208"}]},"ts":"1676290949208"} 2023-02-13 12:22:29,209 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLED in hbase:meta 2023-02-13 12:22:29,243 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676290926065' was successfully enabled. 2023-02-13 12:22:29,246 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676290926065 in 564 msec 2023-02-13 12:22:29,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 12:22:29,437 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 148 completed 2023-02-13 12:22:29,438 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ReadOnlyZKClient(139): Connect 0x2bcbab29 to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:29,452 DEBUG [Listener at localhost.localdomain/46179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62b65b07, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:29,454 DEBUG [hconnection-0x5484563a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:29,458 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:40150, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:29,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,656 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b628e69 to 127.0.0.1:62541 2023-02-13 12:22:29,656 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:29,657 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x408a892a to 127.0.0.1:62541 2023-02-13 12:22:29,657 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:29,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,727 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 12:22:29,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,756 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 12:22:29,763 DEBUG [hconnection-0x5484563a-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:29,770 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45302, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:29,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:29,994 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 12:22:29,997 DEBUG [hconnection-0x5484563a-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:29,999 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:51510, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:30,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,238 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 12:22:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,483 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 12:22:30,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,735 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 12:22:30,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:30,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:31,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:32,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:33,290 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676290926065 2023-02-13 12:22:33,291 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676290926065 2023-02-13 12:22:33,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:33,297 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290953297"}]},"ts":"1676290953297"} 2023-02-13 12:22:33,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-13 12:22:33,298 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLING in hbase:meta 2023-02-13 12:22:33,335 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676290926065 to state=DISABLING 2023-02-13 12:22:33,336 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN}] 2023-02-13 12:22:33,338 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN 2023-02-13 12:22:33,338 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN 2023-02-13 12:22:33,338 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN 2023-02-13 12:22:33,338 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN 2023-02-13 12:22:33,339 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN 2023-02-13 12:22:33,339 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:33,339 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:33,339 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:33,339 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290953339"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290953339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290953339"}]},"ts":"1676290953339"} 2023-02-13 12:22:33,340 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290953339"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290953339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290953339"}]},"ts":"1676290953339"} 2023-02-13 12:22:33,339 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:33,339 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290953339"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290953339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290953339"}]},"ts":"1676290953339"} 2023-02-13 12:22:33,339 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:33,340 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290953339"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290953339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290953339"}]},"ts":"1676290953339"} 2023-02-13 12:22:33,340 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290953339"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290953339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290953339"}]},"ts":"1676290953339"} 2023-02-13 12:22:33,341 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=162, state=RUNNABLE; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:33,342 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=166, state=RUNNABLE; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:33,343 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=164, state=RUNNABLE; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:33,344 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=167, state=RUNNABLE; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:33,345 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN 2023-02-13 12:22:33,345 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=163, state=RUNNABLE; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:33,348 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:33,348 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290953348"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290953348"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290953348"}]},"ts":"1676290953348"} 2023-02-13 12:22:33,350 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=165, state=RUNNABLE; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:33,494 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:33,494 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:33,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 50cc7d0236be777dd672124c8ff93b56, disabling compactions & flushes 2023-02-13 12:22:33,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6e9099611b20e9378ce559b4b3a17188, disabling compactions & flushes 2023-02-13 12:22:33,495 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:33,495 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:33,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:33,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:33,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. after waiting 0 ms 2023-02-13 12:22:33,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. after waiting 0 ms 2023-02-13 12:22:33,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:33,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:33,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:33,501 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 585751c2808ad67a24a61f19e9ac17f5, disabling compactions & flushes 2023-02-13 12:22:33,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:33,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:33,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. after waiting 0 ms 2023-02-13 12:22:33,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:33,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 12:22:33,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 12:22:33,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 12:22:33,512 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:33,512 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:33,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:33,513 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6e9099611b20e9378ce559b4b3a17188: Waiting for close lock at 1676290953494Running coprocessor pre-close hooks at 1676290953494Disabling compacts and flushes for region at 1676290953494Disabling writes for close at 1676290953495 (+1 ms)Writing region close event to WAL at 1676290953500 (+5 ms)Running coprocessor post-close hooks at 1676290953510 (+10 ms)Closed at 1676290953512 (+2 ms) 2023-02-13 12:22:33,513 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 585751c2808ad67a24a61f19e9ac17f5: Waiting for close lock at 1676290953501Running coprocessor pre-close hooks at 1676290953501Disabling compacts and flushes for region at 1676290953501Disabling writes for close at 1676290953502 (+1 ms)Writing region close event to WAL at 1676290953506 (+4 ms)Running coprocessor post-close hooks at 1676290953512 (+6 ms)Closed at 1676290953513 (+1 ms) 2023-02-13 12:22:33,513 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 50cc7d0236be777dd672124c8ff93b56: Waiting for close lock at 1676290953494Running coprocessor pre-close hooks at 1676290953494Disabling compacts and flushes for region at 1676290953494Disabling writes for close at 1676290953495 (+1 ms)Writing region close event to WAL at 1676290953500 (+5 ms)Running coprocessor post-close hooks at 1676290953510 (+10 ms)Closed at 1676290953512 (+2 ms) 2023-02-13 12:22:33,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:33,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:33,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81a0069ff66a08b05e0e5fe4210e0ce2, disabling compactions & flushes 2023-02-13 12:22:33,515 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:33,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:33,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. after waiting 0 ms 2023-02-13 12:22:33,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:33,517 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSED 2023-02-13 12:22:33,521 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290953516"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290953516"}]},"ts":"1676290953516"} 2023-02-13 12:22:33,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:33,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:33,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6d6fd747f58897a134fa0a460eb9ec99, disabling compactions & flushes 2023-02-13 12:22:33,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:33,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:33,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. after waiting 0 ms 2023-02-13 12:22:33,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:33,522 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSED 2023-02-13 12:22:33,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:33,522 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290953522"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290953522"}]},"ts":"1676290953522"} 2023-02-13 12:22:33,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:33,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 197a58e90a6010ab47efab725a975e4c, disabling compactions & flushes 2023-02-13 12:22:33,524 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSED 2023-02-13 12:22:33,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:33,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:33,525 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290953524"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290953524"}]},"ts":"1676290953524"} 2023-02-13 12:22:33,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. after waiting 0 ms 2023-02-13 12:22:33,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:33,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 12:22:33,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:33,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 12:22:33,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Waiting for close lock at 1676290953514Running coprocessor pre-close hooks at 1676290953514Disabling compacts and flushes for region at 1676290953514Disabling writes for close at 1676290953515 (+1 ms)Writing region close event to WAL at 1676290953521 (+6 ms)Running coprocessor post-close hooks at 1676290953537 (+16 ms)Closed at 1676290953537 2023-02-13 12:22:33,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:33,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 12:22:33,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6d6fd747f58897a134fa0a460eb9ec99: Waiting for close lock at 1676290953521Running coprocessor pre-close hooks at 1676290953521Disabling compacts and flushes for region at 1676290953521Disabling writes for close at 1676290953522 (+1 ms)Writing region close event to WAL at 1676290953528 (+6 ms)Running coprocessor post-close hooks at 1676290953537 (+9 ms)Closed at 1676290953537 2023-02-13 12:22:33,538 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:33,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 197a58e90a6010ab47efab725a975e4c: Waiting for close lock at 1676290953523Running coprocessor pre-close hooks at 1676290953523Disabling compacts and flushes for region at 1676290953523Disabling writes for close at 1676290953528 (+5 ms)Writing region close event to WAL at 1676290953533 (+5 ms)Running coprocessor post-close hooks at 1676290953538 (+5 ms)Closed at 1676290953538 2023-02-13 12:22:33,540 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=162 2023-02-13 12:22:33,540 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=162, state=SUCCESS; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,34079,1676290923111 in 193 msec 2023-02-13 12:22:33,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:33,542 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSED 2023-02-13 12:22:33,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:33,543 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290953542"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290953542"}]},"ts":"1676290953542"} 2023-02-13 12:22:33,543 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=167 2023-02-13 12:22:33,543 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=167, state=SUCCESS; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148 in 192 msec 2023-02-13 12:22:33,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN in 204 msec 2023-02-13 12:22:33,544 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSED 2023-02-13 12:22:33,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:33,544 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290953544"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290953544"}]},"ts":"1676290953544"} 2023-02-13 12:22:33,545 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSED 2023-02-13 12:22:33,545 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=163 2023-02-13 12:22:33,545 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290953545"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290953545"}]},"ts":"1676290953545"} 2023-02-13 12:22:33,545 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=163, state=SUCCESS; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34139,1676290923061 in 196 msec 2023-02-13 12:22:33,545 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN in 208 msec 2023-02-13 12:22:33,546 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=165 2023-02-13 12:22:33,546 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN in 209 msec 2023-02-13 12:22:33,546 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=165, state=SUCCESS; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34079,1676290923111 in 194 msec 2023-02-13 12:22:33,547 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN in 210 msec 2023-02-13 12:22:33,549 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=164 2023-02-13 12:22:33,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-13 12:22:33,549 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=164, state=SUCCESS; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,36049,1676290923148 in 204 msec 2023-02-13 12:22:33,549 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=166 2023-02-13 12:22:33,549 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=166, state=SUCCESS; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34139,1676290923061 in 206 msec 2023-02-13 12:22:33,551 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN in 213 msec 2023-02-13 12:22:33,552 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=166, resume processing ppid=161 2023-02-13 12:22:33,552 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN in 213 msec 2023-02-13 12:22:33,552 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290953552"}]},"ts":"1676290953552"} 2023-02-13 12:22:33,553 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLED in hbase:meta 2023-02-13 12:22:33,560 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676290926065 to state=DISABLED 2023-02-13 12:22:33,562 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676290926065 in 270 msec 2023-02-13 12:22:34,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-13 12:22:34,050 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 161 completed 2023-02-13 12:22:34,054 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$5(2215): Client=jenkins//188.40.62.62 delete testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,061 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,063 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-13 12:22:34,070 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:34,070 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:34,070 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:34,070 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:34,070 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:34,070 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:34,072 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits] 2023-02-13 12:22:34,072 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits] 2023-02-13 12:22:34,072 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits] 2023-02-13 12:22:34,073 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits] 2023-02-13 12:22:34,073 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits] 2023-02-13 12:22:34,073 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits] 2023-02-13 12:22:34,076 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-0a65255f37e3420c9fdfa034125fc5c8 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-0a65255f37e3420c9fdfa034125fc5c8 2023-02-13 12:22:34,077 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-05ca7eda38a34d9c85aaeaa016b874da to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-05ca7eda38a34d9c85aaeaa016b874da 2023-02-13 12:22:34,077 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-4446486d315045788982bdebc8a2bb90 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-4446486d315045788982bdebc8a2bb90 2023-02-13 12:22:34,077 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-189c775dee9b461ca79ff1ad57032a6e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-189c775dee9b461ca79ff1ad57032a6e 2023-02-13 12:22:34,077 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-31b72b71ccea46d8a373f1588b857585 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-31b72b71ccea46d8a373f1588b857585 2023-02-13 12:22:34,078 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-68333e5f91e3413fbabd03a7c9da49fa to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-68333e5f91e3413fbabd03a7c9da49fa 2023-02-13 12:22:34,079 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-48186611db074ebeab6e0941597e1c9e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-48186611db074ebeab6e0941597e1c9e 2023-02-13 12:22:34,079 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-3e370cf8b62343469a87e9049f031efb to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-3e370cf8b62343469a87e9049f031efb 2023-02-13 12:22:34,079 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-24b7ea7f7b1d42cfa460732691ead827 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-24b7ea7f7b1d42cfa460732691ead827 2023-02-13 12:22:34,079 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-84c9c4a754f148fc8187a50ff196c186 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-84c9c4a754f148fc8187a50ff196c186 2023-02-13 12:22:34,080 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-d1029a30e4964e55bcbe8195fe1e7315 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-d1029a30e4964e55bcbe8195fe1e7315 2023-02-13 12:22:34,080 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-79d6c194793c4877a70ea1a6905cd846 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-79d6c194793c4877a70ea1a6905cd846 2023-02-13 12:22:34,083 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits/27.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits/27.seqid 2023-02-13 12:22:34,083 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits/27.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits/27.seqid 2023-02-13 12:22:34,084 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:34,084 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits/27.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits/27.seqid 2023-02-13 12:22:34,084 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits/27.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits/27.seqid 2023-02-13 12:22:34,084 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits/27.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits/27.seqid 2023-02-13 12:22:34,084 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:34,084 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:34,084 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:34,084 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:34,088 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits/27.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits/27.seqid 2023-02-13 12:22:34,089 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:34,089 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676290926065 regions 2023-02-13 12:22:34,089 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081 2023-02-13 12:22:34,090 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf] 2023-02-13 12:22:34,091 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a 2023-02-13 12:22:34,092 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c 2023-02-13 12:22:34,093 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 2023-02-13 12:22:34,094 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 2023-02-13 12:22:34,095 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c 2023-02-13 12:22:34,095 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af 2023-02-13 12:22:34,096 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e 2023-02-13 12:22:34,097 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f 2023-02-13 12:22:34,098 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 2023-02-13 12:22:34,098 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 2023-02-13 12:22:34,099 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 2023-02-13 12:22:34,100 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f 2023-02-13 12:22:34,100 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081 2023-02-13 12:22:34,102 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,122 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676290926065 from hbase:meta 2023-02-13 12:22:34,125 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676290926065' descriptor. 2023-02-13 12:22:34,126 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,126 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676290926065' from region states. 2023-02-13 12:22:34,127 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290954127"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:34,127 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290954127"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:34,127 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290954127"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:34,127 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290954127"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:34,127 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290954127"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:34,127 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290954127"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:34,129 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 12:22:34,129 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 12:22:34,129 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676290926065' as deleted. 2023-02-13 12:22:34,129 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676290954129"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:34,131 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676290926065 state from META 2023-02-13 12:22:34,140 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,142 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 in 86 msec 2023-02-13 12:22:34,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-13 12:22:34,315 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 174 completed 2023-02-13 12:22:34,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676290926065 snapshot=name: "snaptb1-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290938618 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 12:22:34,364 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676290926065 as table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 12:22:34,377 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/.tabledesc/.tableinfo.0000000001 2023-02-13 12:22:34,378 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290938618 type: DISABLED version: 2 ttl: 0 2023-02-13 12:22:34,378 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,379 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,381 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,383 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,385 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,386 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,388 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,389 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,393 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,395 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,397 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,399 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,400 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,402 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:34,402 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:34,402 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:34,402 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:34,402 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:34,402 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:34,402 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=585751c2808ad67a24a61f19e9ac17f5 as 585751c2808ad67a24a61f19e9ac17f5 in snapshot snaptb1-1676290926065 2023-02-13 12:22:34,402 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=50cc7d0236be777dd672124c8ff93b56 as 50cc7d0236be777dd672124c8ff93b56 in snapshot snaptb1-1676290926065 2023-02-13 12:22:34,402 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=197a58e90a6010ab47efab725a975e4c as 197a58e90a6010ab47efab725a975e4c in snapshot snaptb1-1676290926065 2023-02-13 12:22:34,402 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=6d6fd747f58897a134fa0a460eb9ec99 as 6d6fd747f58897a134fa0a460eb9ec99 in snapshot snaptb1-1676290926065 2023-02-13 12:22:34,402 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=81a0069ff66a08b05e0e5fe4210e0ce2 as 81a0069ff66a08b05e0e5fe4210e0ce2 in snapshot snaptb1-1676290926065 2023-02-13 12:22:34,402 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=6e9099611b20e9378ce559b4b3a17188 as 6e9099611b20e9378ce559b4b3a17188 in snapshot snaptb1-1676290926065 2023-02-13 12:22:34,403 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676290926065', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:34,404 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676290926065', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:34,404 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676290926065', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:34,404 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676290926065', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:34,404 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676290926065', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:34,404 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676290926065', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:34,432 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:34,432 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 4446486d315045788982bdebc8a2bb90 from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,440 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:34,440 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 31b72b71ccea46d8a373f1588b857585 from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,445 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:34,445 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 05ca7eda38a34d9c85aaeaa016b874da from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,452 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 84c9c4a754f148fc8187a50ff196c186 from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,455 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:34,455 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:34,455 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 189c775dee9b461ca79ff1ad57032a6e from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,455 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 68333e5f91e3413fbabd03a7c9da49fa from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,456 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 3e370cf8b62343469a87e9049f031efb from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,459 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d1029a30e4964e55bcbe8195fe1e7315 from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,459 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 24b7ea7f7b1d42cfa460732691ead827 from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,459 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing 50cc7d0236be777dd672124c8ff93b56, disabling compactions & flushes 2023-02-13 12:22:34,459 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:34,459 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:34,459 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. after waiting 0 ms 2023-02-13 12:22:34,459 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:34,459 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:34,459 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for 50cc7d0236be777dd672124c8ff93b56: Waiting for close lock at 1676290954459Disabling compacts and flushes for region at 1676290954459Disabling writes for close at 1676290954459Writing region close event to WAL at 1676290954459Closed at 1676290954459 2023-02-13 12:22:34,460 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 79d6c194793c4877a70ea1a6905cd846 from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,460 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing 81a0069ff66a08b05e0e5fe4210e0ce2, disabling compactions & flushes 2023-02-13 12:22:34,460 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:34,460 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:34,460 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. after waiting 0 ms 2023-02-13 12:22:34,460 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:34,460 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:34,460 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Waiting for close lock at 1676290954460Disabling compacts and flushes for region at 1676290954460Disabling writes for close at 1676290954460Writing region close event to WAL at 1676290954460Closed at 1676290954460 2023-02-13 12:22:34,463 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:34,463 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 0a65255f37e3420c9fdfa034125fc5c8 from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,464 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing 585751c2808ad67a24a61f19e9ac17f5, disabling compactions & flushes 2023-02-13 12:22:34,464 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:34,464 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing 6d6fd747f58897a134fa0a460eb9ec99, disabling compactions & flushes 2023-02-13 12:22:34,464 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing 197a58e90a6010ab47efab725a975e4c, disabling compactions & flushes 2023-02-13 12:22:34,464 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:34,464 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:34,464 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:34,465 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:34,465 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. after waiting 0 ms 2023-02-13 12:22:34,465 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. after waiting 0 ms 2023-02-13 12:22:34,465 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:34,465 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:34,465 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:34,465 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:34,465 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for 197a58e90a6010ab47efab725a975e4c: Waiting for close lock at 1676290954464Disabling compacts and flushes for region at 1676290954464Disabling writes for close at 1676290954465 (+1 ms)Writing region close event to WAL at 1676290954465Closed at 1676290954465 2023-02-13 12:22:34,465 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. after waiting 0 ms 2023-02-13 12:22:34,465 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:34,465 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:34,465 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for 585751c2808ad67a24a61f19e9ac17f5: Waiting for close lock at 1676290954464Disabling compacts and flushes for region at 1676290954464Disabling writes for close at 1676290954465 (+1 ms)Writing region close event to WAL at 1676290954465Closed at 1676290954465 2023-02-13 12:22:34,465 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:34,465 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for 6d6fd747f58897a134fa0a460eb9ec99: Waiting for close lock at 1676290954464Disabling compacts and flushes for region at 1676290954464Disabling writes for close at 1676290954465 (+1 ms)Writing region close event to WAL at 1676290954465Closed at 1676290954465 2023-02-13 12:22:34,465 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 48186611db074ebeab6e0941597e1c9e from cloned region in snapshot snaptb1-1676290926065 to table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:34,467 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing 6e9099611b20e9378ce559b4b3a17188, disabling compactions & flushes 2023-02-13 12:22:34,467 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:34,467 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:34,467 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. after waiting 0 ms 2023-02-13 12:22:34,467 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:34,467 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:34,467 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for 6e9099611b20e9378ce559b4b3a17188: Waiting for close lock at 1676290954467Disabling compacts and flushes for region at 1676290954467Disabling writes for close at 1676290954467Writing region close event to WAL at 1676290954467Closed at 1676290954467 2023-02-13 12:22:34,467 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290938618 type: DISABLED version: 2 ttl: 0 2023-02-13 12:22:34,467 INFO [PEWorker-3] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676290926065 on table=testRestoreSnapshot-1676290926065 completed! 2023-02-13 12:22:34,471 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290954471"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290954471"}]},"ts":"1676290954471"} 2023-02-13 12:22:34,471 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290954471"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290954471"}]},"ts":"1676290954471"} 2023-02-13 12:22:34,471 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290954471"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290954471"}]},"ts":"1676290954471"} 2023-02-13 12:22:34,471 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290954471"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290954471"}]},"ts":"1676290954471"} 2023-02-13 12:22:34,471 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290954471"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290954471"}]},"ts":"1676290954471"} 2023-02-13 12:22:34,471 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290954471"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290954471"}]},"ts":"1676290954471"} 2023-02-13 12:22:34,473 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 12:22:34,474 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290954473"}]},"ts":"1676290954473"} 2023-02-13 12:22:34,474 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLING in hbase:meta 2023-02-13 12:22:34,495 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 12:22:34,495 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 12:22:34,495 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 12:22:34,495 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 12:22:34,495 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 12:22:34,495 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN}] 2023-02-13 12:22:34,498 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN 2023-02-13 12:22:34,498 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN 2023-02-13 12:22:34,498 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN 2023-02-13 12:22:34,498 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN 2023-02-13 12:22:34,498 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=false 2023-02-13 12:22:34,498 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=false 2023-02-13 12:22:34,498 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN 2023-02-13 12:22:34,499 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=false 2023-02-13 12:22:34,499 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=false 2023-02-13 12:22:34,499 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN 2023-02-13 12:22:34,500 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=false 2023-02-13 12:22:34,500 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=false 2023-02-13 12:22:34,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 12:22:34,628 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 12:22:34,649 INFO [jenkins-hbase16:46677] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 12:22:34,653 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:34,653 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:34,653 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:34,653 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290954653"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290954653"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290954653"}]},"ts":"1676290954653"} 2023-02-13 12:22:34,653 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:34,653 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:34,653 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290954653"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290954653"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290954653"}]},"ts":"1676290954653"} 2023-02-13 12:22:34,653 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290954653"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290954653"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290954653"}]},"ts":"1676290954653"} 2023-02-13 12:22:34,653 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290954653"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290954653"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290954653"}]},"ts":"1676290954653"} 2023-02-13 12:22:34,653 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290954653"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290954653"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290954653"}]},"ts":"1676290954653"} 2023-02-13 12:22:34,655 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=179, state=RUNNABLE; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:34,658 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=180, state=RUNNABLE; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:34,658 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=176, state=RUNNABLE; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:34,660 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=178, state=RUNNABLE; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:34,660 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=181, state=RUNNABLE; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:34,660 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:34,660 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290954660"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290954660"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290954660"}]},"ts":"1676290954660"} 2023-02-13 12:22:34,662 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=177, state=RUNNABLE; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:34,811 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:34,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:22:34,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:34,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:34,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:34,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:34,813 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:34,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:22:34,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:34,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:34,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:34,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:34,813 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:34,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:22:34,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:34,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:34,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:34,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:34,814 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:34,815 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:34,815 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:34,815 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:34,815 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf 2023-02-13 12:22:34,815 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:34,815 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf 2023-02-13 12:22:34,815 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 50cc7d0236be777dd672124c8ff93b56 columnFamilyName cf 2023-02-13 12:22:34,815 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:34,815 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf 2023-02-13 12:22:34,815 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 197a58e90a6010ab47efab725a975e4c columnFamilyName cf 2023-02-13 12:22:34,816 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d6fd747f58897a134fa0a460eb9ec99 columnFamilyName cf 2023-02-13 12:22:34,825 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-4446486d315045788982bdebc8a2bb90 2023-02-13 12:22:34,825 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-189c775dee9b461ca79ff1ad57032a6e 2023-02-13 12:22:34,825 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-05ca7eda38a34d9c85aaeaa016b874da 2023-02-13 12:22:34,835 DEBUG [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-3e370cf8b62343469a87e9049f031efb 2023-02-13 12:22:34,836 INFO [StoreOpener-50cc7d0236be777dd672124c8ff93b56-1] regionserver.HStore(310): Store=50cc7d0236be777dd672124c8ff93b56/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:34,836 DEBUG [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-24b7ea7f7b1d42cfa460732691ead827 2023-02-13 12:22:34,836 INFO [StoreOpener-197a58e90a6010ab47efab725a975e4c-1] regionserver.HStore(310): Store=197a58e90a6010ab47efab725a975e4c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:34,837 DEBUG [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-d1029a30e4964e55bcbe8195fe1e7315 2023-02-13 12:22:34,837 INFO [StoreOpener-6d6fd747f58897a134fa0a460eb9ec99-1] regionserver.HStore(310): Store=6d6fd747f58897a134fa0a460eb9ec99/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:34,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:34,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:34,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:34,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:34,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:34,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:34,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:34,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:34,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:34,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 12:22:34,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 12:22:34,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 12:22:34,852 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 50cc7d0236be777dd672124c8ff93b56; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62988154, jitterRate=-0.061403363943099976}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:34,852 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 197a58e90a6010ab47efab725a975e4c; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73584877, jitterRate=0.09650011360645294}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:34,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 197a58e90a6010ab47efab725a975e4c: Running coprocessor pre-open hook at 1676290954813Writing region info on filesystem at 1676290954813Initializing all the Stores at 1676290954814 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290954814Cleaning up temporary data from old regions at 1676290954844 (+30 ms)Cleaning up detritus from prior splits at 1676290954844Running coprocessor post-open hooks at 1676290954852 (+8 ms)Region opened successfully at 1676290954852 2023-02-13 12:22:34,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 50cc7d0236be777dd672124c8ff93b56: Running coprocessor pre-open hook at 1676290954813Writing region info on filesystem at 1676290954813Initializing all the Stores at 1676290954814 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290954814Cleaning up temporary data from old regions at 1676290954840 (+26 ms)Cleaning up detritus from prior splits at 1676290954844 (+4 ms)Running coprocessor post-open hooks at 1676290954852 (+8 ms)Region opened successfully at 1676290954852 2023-02-13 12:22:34,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56., pid=187, masterSystemTime=1676290954809 2023-02-13 12:22:34,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6d6fd747f58897a134fa0a460eb9ec99; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67601632, jitterRate=0.007342815399169922}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:34,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c., pid=183, masterSystemTime=1676290954810 2023-02-13 12:22:34,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6d6fd747f58897a134fa0a460eb9ec99: Running coprocessor pre-open hook at 1676290954814Writing region info on filesystem at 1676290954814Initializing all the Stores at 1676290954814Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290954815 (+1 ms)Cleaning up temporary data from old regions at 1676290954844 (+29 ms)Cleaning up detritus from prior splits at 1676290954845 (+1 ms)Running coprocessor post-open hooks at 1676290954854 (+9 ms)Region opened successfully at 1676290954854 2023-02-13 12:22:34,854 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99., pid=185, masterSystemTime=1676290954811 2023-02-13 12:22:34,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:34,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:34,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:34,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:22:34,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:34,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:34,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:34,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:34,862 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:34,862 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290954862"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290954862"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290954862"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290954862"}]},"ts":"1676290954862"} 2023-02-13 12:22:34,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:34,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:34,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:34,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:22:34,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:34,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:34,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:34,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:34,864 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:34,864 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290954864"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290954864"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290954864"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290954864"}]},"ts":"1676290954864"} 2023-02-13 12:22:34,865 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:34,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:34,866 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:34,866 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:34,866 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:34,866 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf 2023-02-13 12:22:34,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:22:34,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676290926065 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:34,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:34,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:34,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:34,866 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81a0069ff66a08b05e0e5fe4210e0ce2 columnFamilyName cf 2023-02-13 12:22:34,867 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:34,867 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290954867"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290954867"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290954867"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290954867"}]},"ts":"1676290954867"} 2023-02-13 12:22:34,867 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:34,869 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:34,869 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf 2023-02-13 12:22:34,869 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6e9099611b20e9378ce559b4b3a17188 columnFamilyName cf 2023-02-13 12:22:34,872 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:34,873 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=177 2023-02-13 12:22:34,873 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=177, state=SUCCESS; OpenRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34079,1676290923111 in 202 msec 2023-02-13 12:22:34,873 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=180 2023-02-13 12:22:34,873 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=180, state=SUCCESS; OpenRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34139,1676290923061 in 210 msec 2023-02-13 12:22:34,874 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=178 2023-02-13 12:22:34,874 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, ASSIGN in 378 msec 2023-02-13 12:22:34,874 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=178, state=SUCCESS; OpenRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,36049,1676290923148 in 210 msec 2023-02-13 12:22:34,876 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, ASSIGN in 378 msec 2023-02-13 12:22:34,876 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, ASSIGN in 379 msec 2023-02-13 12:22:34,880 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:34,880 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf 2023-02-13 12:22:34,881 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 585751c2808ad67a24a61f19e9ac17f5 columnFamilyName cf 2023-02-13 12:22:34,893 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-31b72b71ccea46d8a373f1588b857585 2023-02-13 12:22:34,894 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-0a65255f37e3420c9fdfa034125fc5c8 2023-02-13 12:22:34,896 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-68333e5f91e3413fbabd03a7c9da49fa 2023-02-13 12:22:34,897 DEBUG [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-84c9c4a754f148fc8187a50ff196c186 2023-02-13 12:22:34,898 INFO [StoreOpener-81a0069ff66a08b05e0e5fe4210e0ce2-1] regionserver.HStore(310): Store=81a0069ff66a08b05e0e5fe4210e0ce2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:34,898 DEBUG [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-48186611db074ebeab6e0941597e1c9e 2023-02-13 12:22:34,898 INFO [StoreOpener-6e9099611b20e9378ce559b4b3a17188-1] regionserver.HStore(310): Store=6e9099611b20e9378ce559b4b3a17188/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:34,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:34,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:34,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:34,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:34,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:34,901 DEBUG [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-79d6c194793c4877a70ea1a6905cd846 2023-02-13 12:22:34,901 INFO [StoreOpener-585751c2808ad67a24a61f19e9ac17f5-1] regionserver.HStore(310): Store=585751c2808ad67a24a61f19e9ac17f5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:34,902 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:34,902 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:34,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:34,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:34,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 12:22:34,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 12:22:34,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 12:22:34,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 585751c2808ad67a24a61f19e9ac17f5; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64999799, jitterRate=-0.03142751753330231}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:34,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 6e9099611b20e9378ce559b4b3a17188; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66484726, jitterRate=-0.009300380945205688}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:34,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 585751c2808ad67a24a61f19e9ac17f5: Running coprocessor pre-open hook at 1676290954866Writing region info on filesystem at 1676290954866Initializing all the Stores at 1676290954867 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290954868 (+1 ms)Cleaning up temporary data from old regions at 1676290954902 (+34 ms)Cleaning up detritus from prior splits at 1676290954902Running coprocessor post-open hooks at 1676290954912 (+10 ms)Region opened successfully at 1676290954912 2023-02-13 12:22:34,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 81a0069ff66a08b05e0e5fe4210e0ce2; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73617559, jitterRate=0.09698711335659027}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:34,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 6e9099611b20e9378ce559b4b3a17188: Running coprocessor pre-open hook at 1676290954863Writing region info on filesystem at 1676290954863Initializing all the Stores at 1676290954866 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290954866Cleaning up temporary data from old regions at 1676290954901 (+35 ms)Cleaning up detritus from prior splits at 1676290954901Running coprocessor post-open hooks at 1676290954912 (+11 ms)Region opened successfully at 1676290954912 2023-02-13 12:22:34,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Running coprocessor pre-open hook at 1676290954861Writing region info on filesystem at 1676290954861Initializing all the Stores at 1676290954865 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290954865Cleaning up temporary data from old regions at 1676290954899 (+34 ms)Cleaning up detritus from prior splits at 1676290954899Running coprocessor post-open hooks at 1676290954912 (+13 ms)Region opened successfully at 1676290954912 2023-02-13 12:22:34,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5., pid=186, masterSystemTime=1676290954811 2023-02-13 12:22:34,913 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2., pid=182, masterSystemTime=1676290954809 2023-02-13 12:22:34,913 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188., pid=184, masterSystemTime=1676290954810 2023-02-13 12:22:34,914 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:34,914 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:34,914 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:34,914 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290954914"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290954914"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290954914"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290954914"}]},"ts":"1676290954914"} 2023-02-13 12:22:34,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:34,915 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:34,915 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:34,915 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290954915"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290954915"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290954915"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290954915"}]},"ts":"1676290954915"} 2023-02-13 12:22:34,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:34,916 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:34,916 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:34,916 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290954916"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290954916"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290954916"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290954916"}]},"ts":"1676290954916"} 2023-02-13 12:22:34,920 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=181 2023-02-13 12:22:34,920 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=181, state=SUCCESS; OpenRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148 in 256 msec 2023-02-13 12:22:34,920 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=176 2023-02-13 12:22:34,920 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=179 2023-02-13 12:22:34,921 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=176, state=SUCCESS; OpenRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,34139,1676290923061 in 259 msec 2023-02-13 12:22:34,921 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=179, state=SUCCESS; OpenRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34079,1676290923111 in 262 msec 2023-02-13 12:22:34,921 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, ASSIGN in 425 msec 2023-02-13 12:22:34,922 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, ASSIGN in 426 msec 2023-02-13 12:22:34,922 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=176, resume processing ppid=175 2023-02-13 12:22:34,922 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, ASSIGN in 426 msec 2023-02-13 12:22:34,923 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290954923"}]},"ts":"1676290954923"} 2023-02-13 12:22:34,926 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=ENABLED in hbase:meta 2023-02-13 12:22:34,963 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676290926065 snapshot=name: "snaptb1-1676290926065" table: "testRestoreSnapshot-1676290926065" creation_time: 1676290938618 type: DISABLED version: 2 ttl: 0 ) in 602 msec 2023-02-13 12:22:34,995 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676290926065' 2023-02-13 12:22:35,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 12:22:35,120 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676290926065, procId: 175 completed 2023-02-13 12:22:35,122 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ReadOnlyZKClient(139): Connect 0x45bdc0c5 to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:35,141 DEBUG [Listener at localhost.localdomain/46179] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@384150f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:35,144 DEBUG [hconnection-0x775e0135-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:35,145 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:38584, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:35,150 DEBUG [hconnection-0x775e0135-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:35,152 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:50882, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:35,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,418 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 12:22:35,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,704 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 12:22:35,710 DEBUG [hconnection-0x775e0135-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:35,711 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:51520, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:35,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,930 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 12:22:35,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:35,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,109 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 12:22:36,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,308 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 12:22:36,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,961 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2bcbab29 to 127.0.0.1:62541 2023-02-13 12:22:36,961 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:36,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:36,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:37,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:38,546 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676290926065 2023-02-13 12:22:38,547 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testRestoreSnapshot-1676290926065 2023-02-13 12:22:38,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:38,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-13 12:22:38,550 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290958550"}]},"ts":"1676290958550"} 2023-02-13 12:22:38,551 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLING in hbase:meta 2023-02-13 12:22:38,559 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676290926065 to state=DISABLING 2023-02-13 12:22:38,560 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN}] 2023-02-13 12:22:38,562 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN 2023-02-13 12:22:38,562 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN 2023-02-13 12:22:38,562 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN 2023-02-13 12:22:38,563 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN 2023-02-13 12:22:38,563 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN 2023-02-13 12:22:38,563 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:38,563 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290958563"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290958563"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290958563"}]},"ts":"1676290958563"} 2023-02-13 12:22:38,563 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:38,563 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:38,563 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290958563"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290958563"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290958563"}]},"ts":"1676290958563"} 2023-02-13 12:22:38,563 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290958563"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290958563"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290958563"}]},"ts":"1676290958563"} 2023-02-13 12:22:38,563 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:38,563 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:38,563 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290958563"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290958563"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290958563"}]},"ts":"1676290958563"} 2023-02-13 12:22:38,564 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290958563"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290958563"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290958563"}]},"ts":"1676290958563"} 2023-02-13 12:22:38,564 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=191, state=RUNNABLE; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:38,565 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=190, state=RUNNABLE; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:38,566 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=189, state=RUNNABLE; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:38,569 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=194, state=RUNNABLE; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:38,571 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=192, state=RUNNABLE; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:38,571 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN 2023-02-13 12:22:38,572 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:38,573 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290958572"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290958572"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290958572"}]},"ts":"1676290958572"} 2023-02-13 12:22:38,574 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=193, state=RUNNABLE; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:38,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:38,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6d6fd747f58897a134fa0a460eb9ec99, disabling compactions & flushes 2023-02-13 12:22:38,719 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:38,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:38,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. after waiting 0 ms 2023-02-13 12:22:38,719 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:38,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:38,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:38,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 81a0069ff66a08b05e0e5fe4210e0ce2, disabling compactions & flushes 2023-02-13 12:22:38,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:38,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:38,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. after waiting 0 ms 2023-02-13 12:22:38,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:38,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 6e9099611b20e9378ce559b4b3a17188, disabling compactions & flushes 2023-02-13 12:22:38,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:38,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:38,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. after waiting 0 ms 2023-02-13 12:22:38,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:38,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 12:22:38,739 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99. 2023-02-13 12:22:38,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6d6fd747f58897a134fa0a460eb9ec99: Waiting for close lock at 1676290958718Running coprocessor pre-close hooks at 1676290958718Disabling compacts and flushes for region at 1676290958718Disabling writes for close at 1676290958719 (+1 ms)Writing region close event to WAL at 1676290958725 (+6 ms)Running coprocessor post-close hooks at 1676290958739 (+14 ms)Closed at 1676290958739 2023-02-13 12:22:38,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 12:22:38,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 12:22:38,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2. 2023-02-13 12:22:38,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 81a0069ff66a08b05e0e5fe4210e0ce2: Waiting for close lock at 1676290958726Running coprocessor pre-close hooks at 1676290958726Disabling compacts and flushes for region at 1676290958726Disabling writes for close at 1676290958727 (+1 ms)Writing region close event to WAL at 1676290958734 (+7 ms)Running coprocessor post-close hooks at 1676290958740 (+6 ms)Closed at 1676290958740 2023-02-13 12:22:38,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188. 2023-02-13 12:22:38,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 6e9099611b20e9378ce559b4b3a17188: Waiting for close lock at 1676290958727Running coprocessor pre-close hooks at 1676290958727Disabling compacts and flushes for region at 1676290958727Disabling writes for close at 1676290958728 (+1 ms)Writing region close event to WAL at 1676290958736 (+8 ms)Running coprocessor post-close hooks at 1676290958740 (+4 ms)Closed at 1676290958740 2023-02-13 12:22:38,741 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:38,741 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:38,742 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 585751c2808ad67a24a61f19e9ac17f5, disabling compactions & flushes 2023-02-13 12:22:38,742 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:38,742 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:38,742 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. after waiting 0 ms 2023-02-13 12:22:38,742 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=6d6fd747f58897a134fa0a460eb9ec99, regionState=CLOSED 2023-02-13 12:22:38,742 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:38,742 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290958742"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290958742"}]},"ts":"1676290958742"} 2023-02-13 12:22:38,742 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:38,742 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:38,743 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=81a0069ff66a08b05e0e5fe4210e0ce2, regionState=CLOSED 2023-02-13 12:22:38,743 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290958743"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290958743"}]},"ts":"1676290958743"} 2023-02-13 12:22:38,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:38,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:38,743 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=6e9099611b20e9378ce559b4b3a17188, regionState=CLOSED 2023-02-13 12:22:38,743 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290958743"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290958743"}]},"ts":"1676290958743"} 2023-02-13 12:22:38,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 197a58e90a6010ab47efab725a975e4c, disabling compactions & flushes 2023-02-13 12:22:38,746 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:38,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:38,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 50cc7d0236be777dd672124c8ff93b56, disabling compactions & flushes 2023-02-13 12:22:38,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. after waiting 0 ms 2023-02-13 12:22:38,746 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:38,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:38,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:38,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. after waiting 3 ms 2023-02-13 12:22:38,753 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=191 2023-02-13 12:22:38,753 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=191, state=SUCCESS; CloseRegionProcedure 6d6fd747f58897a134fa0a460eb9ec99, server=jenkins-hbase16.apache.org,36049,1676290923148 in 179 msec 2023-02-13 12:22:38,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:38,754 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=189 2023-02-13 12:22:38,755 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=189, state=SUCCESS; CloseRegionProcedure 6e9099611b20e9378ce559b4b3a17188, server=jenkins-hbase16.apache.org,34139,1676290923061 in 179 msec 2023-02-13 12:22:38,755 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=192 2023-02-13 12:22:38,755 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=192, state=SUCCESS; CloseRegionProcedure 81a0069ff66a08b05e0e5fe4210e0ce2, server=jenkins-hbase16.apache.org,34079,1676290923111 in 173 msec 2023-02-13 12:22:38,756 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6d6fd747f58897a134fa0a460eb9ec99, UNASSIGN in 193 msec 2023-02-13 12:22:38,756 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 12:22:38,757 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=6e9099611b20e9378ce559b4b3a17188, UNASSIGN in 195 msec 2023-02-13 12:22:38,757 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=81a0069ff66a08b05e0e5fe4210e0ce2, UNASSIGN in 195 msec 2023-02-13 12:22:38,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5. 2023-02-13 12:22:38,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 585751c2808ad67a24a61f19e9ac17f5: Waiting for close lock at 1676290958741Running coprocessor pre-close hooks at 1676290958741Disabling compacts and flushes for region at 1676290958741Disabling writes for close at 1676290958742 (+1 ms)Writing region close event to WAL at 1676290958753 (+11 ms)Running coprocessor post-close hooks at 1676290958757 (+4 ms)Closed at 1676290958757 2023-02-13 12:22:38,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 12:22:38,758 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c. 2023-02-13 12:22:38,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 197a58e90a6010ab47efab725a975e4c: Waiting for close lock at 1676290958743Running coprocessor pre-close hooks at 1676290958743Disabling compacts and flushes for region at 1676290958743Disabling writes for close at 1676290958746 (+3 ms)Writing region close event to WAL at 1676290958753 (+7 ms)Running coprocessor post-close hooks at 1676290958758 (+5 ms)Closed at 1676290958758 2023-02-13 12:22:38,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 12:22:38,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:38,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56. 2023-02-13 12:22:38,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 50cc7d0236be777dd672124c8ff93b56: Waiting for close lock at 1676290958742Running coprocessor pre-close hooks at 1676290958742Disabling compacts and flushes for region at 1676290958742Disabling writes for close at 1676290958753 (+11 ms)Writing region close event to WAL at 1676290958754 (+1 ms)Running coprocessor post-close hooks at 1676290958760 (+6 ms)Closed at 1676290958760 2023-02-13 12:22:38,760 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=585751c2808ad67a24a61f19e9ac17f5, regionState=CLOSED 2023-02-13 12:22:38,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:38,760 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676290958760"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290958760"}]},"ts":"1676290958760"} 2023-02-13 12:22:38,761 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=197a58e90a6010ab47efab725a975e4c, regionState=CLOSED 2023-02-13 12:22:38,761 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290958761"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290958761"}]},"ts":"1676290958761"} 2023-02-13 12:22:38,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:38,761 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=50cc7d0236be777dd672124c8ff93b56, regionState=CLOSED 2023-02-13 12:22:38,761 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676290958761"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290958761"}]},"ts":"1676290958761"} 2023-02-13 12:22:38,764 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=194 2023-02-13 12:22:38,764 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=194, state=SUCCESS; CloseRegionProcedure 585751c2808ad67a24a61f19e9ac17f5, server=jenkins-hbase16.apache.org,36049,1676290923148 in 194 msec 2023-02-13 12:22:38,765 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=193 2023-02-13 12:22:38,765 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=193, state=SUCCESS; CloseRegionProcedure 197a58e90a6010ab47efab725a975e4c, server=jenkins-hbase16.apache.org,34139,1676290923061 in 188 msec 2023-02-13 12:22:38,766 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=585751c2808ad67a24a61f19e9ac17f5, UNASSIGN in 204 msec 2023-02-13 12:22:38,766 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=190 2023-02-13 12:22:38,766 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=190, state=SUCCESS; CloseRegionProcedure 50cc7d0236be777dd672124c8ff93b56, server=jenkins-hbase16.apache.org,34079,1676290923111 in 198 msec 2023-02-13 12:22:38,766 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=197a58e90a6010ab47efab725a975e4c, UNASSIGN in 205 msec 2023-02-13 12:22:38,767 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=190, resume processing ppid=188 2023-02-13 12:22:38,767 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676290926065, region=50cc7d0236be777dd672124c8ff93b56, UNASSIGN in 206 msec 2023-02-13 12:22:38,768 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290958768"}]},"ts":"1676290958768"} 2023-02-13 12:22:38,769 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676290926065, state=DISABLED in hbase:meta 2023-02-13 12:22:38,776 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676290926065 to state=DISABLED 2023-02-13 12:22:38,778 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676290926065 in 230 msec 2023-02-13 12:22:38,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-13 12:22:38,801 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 188 completed 2023-02-13 12:22:38,802 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$5(2215): Client=jenkins//188.40.62.62 delete testRestoreSnapshot-1676290926065 2023-02-13 12:22:38,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:38,805 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:38,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-13 12:22:38,806 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:38,812 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:38,813 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:38,813 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:38,813 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:38,813 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:38,813 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:38,815 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits] 2023-02-13 12:22:38,815 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits] 2023-02-13 12:22:38,815 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits] 2023-02-13 12:22:38,815 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits] 2023-02-13 12:22:38,815 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits] 2023-02-13 12:22:38,816 DEBUG [HFileArchiver-10] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits] 2023-02-13 12:22:38,821 ERROR [HFileArchiver-10] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-05ca7eda38a34d9c85aaeaa016b874da already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-05ca7eda38a34d9c85aaeaa016b874da. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954076 currentMtime: 1676290954455 2023-02-13 12:22:38,822 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-189c775dee9b461ca79ff1ad57032a6e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-189c775dee9b461ca79ff1ad57032a6e. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954075 currentMtime: 1676290954458 2023-02-13 12:22:38,822 INFO [HFileArchiver-10] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-05ca7eda38a34d9c85aaeaa016b874da to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-05ca7eda38a34d9c85aaeaa016b874da.1676290958816. 2023-02-13 12:22:38,822 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-4446486d315045788982bdebc8a2bb90 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-4446486d315045788982bdebc8a2bb90. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954076 currentMtime: 1676290954452 2023-02-13 12:22:38,822 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-68333e5f91e3413fbabd03a7c9da49fa already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-68333e5f91e3413fbabd03a7c9da49fa. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954076 currentMtime: 1676290954459 2023-02-13 12:22:38,822 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-4446486d315045788982bdebc8a2bb90 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-4446486d315045788982bdebc8a2bb90.1676290958815. 2023-02-13 12:22:38,822 ERROR [HFileArchiver-9] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-0a65255f37e3420c9fdfa034125fc5c8 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-0a65255f37e3420c9fdfa034125fc5c8. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954075 currentMtime: 1676290954465 2023-02-13 12:22:38,822 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-31b72b71ccea46d8a373f1588b857585 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-31b72b71ccea46d8a373f1588b857585. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954076 currentMtime: 1676290954448 2023-02-13 12:22:38,823 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-189c775dee9b461ca79ff1ad57032a6e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-189c775dee9b461ca79ff1ad57032a6e.1676290958815. 2023-02-13 12:22:38,823 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-68333e5f91e3413fbabd03a7c9da49fa to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-68333e5f91e3413fbabd03a7c9da49fa.1676290958815. 2023-02-13 12:22:38,823 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-05ca7eda38a34d9c85aaeaa016b874da to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-05ca7eda38a34d9c85aaeaa016b874da 2023-02-13 12:22:38,824 INFO [HFileArchiver-9] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-0a65255f37e3420c9fdfa034125fc5c8 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-0a65255f37e3420c9fdfa034125fc5c8.1676290958815. 2023-02-13 12:22:38,824 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-31b72b71ccea46d8a373f1588b857585 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-31b72b71ccea46d8a373f1588b857585.1676290958815. 2023-02-13 12:22:38,824 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-4446486d315045788982bdebc8a2bb90 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-4446486d315045788982bdebc8a2bb90 2023-02-13 12:22:38,824 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-68333e5f91e3413fbabd03a7c9da49fa to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-68333e5f91e3413fbabd03a7c9da49fa 2023-02-13 12:22:38,825 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-189c775dee9b461ca79ff1ad57032a6e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-189c775dee9b461ca79ff1ad57032a6e 2023-02-13 12:22:38,825 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-31b72b71ccea46d8a373f1588b857585 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-31b72b71ccea46d8a373f1588b857585 2023-02-13 12:22:38,825 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-0a65255f37e3420c9fdfa034125fc5c8 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-0a65255f37e3420c9fdfa034125fc5c8 2023-02-13 12:22:38,827 ERROR [HFileArchiver-10] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-3e370cf8b62343469a87e9049f031efb already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-3e370cf8b62343469a87e9049f031efb. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954078 currentMtime: 1676290954458 2023-02-13 12:22:38,828 INFO [HFileArchiver-10] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-3e370cf8b62343469a87e9049f031efb to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-3e370cf8b62343469a87e9049f031efb.1676290958816. 2023-02-13 12:22:38,828 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-d1029a30e4964e55bcbe8195fe1e7315 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-d1029a30e4964e55bcbe8195fe1e7315. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954078 currentMtime: 1676290954464 2023-02-13 12:22:38,828 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-d1029a30e4964e55bcbe8195fe1e7315 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-d1029a30e4964e55bcbe8195fe1e7315.1676290958815. 2023-02-13 12:22:38,828 ERROR [HFileArchiver-9] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-48186611db074ebeab6e0941597e1c9e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-48186611db074ebeab6e0941597e1c9e. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954077 currentMtime: 1676290954467 2023-02-13 12:22:38,829 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-79d6c194793c4877a70ea1a6905cd846 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-79d6c194793c4877a70ea1a6905cd846. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954079 currentMtime: 1676290954464 2023-02-13 12:22:38,829 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-84c9c4a754f148fc8187a50ff196c186 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-84c9c4a754f148fc8187a50ff196c186. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954078 currentMtime: 1676290954460 2023-02-13 12:22:38,829 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-24b7ea7f7b1d42cfa460732691ead827 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-24b7ea7f7b1d42cfa460732691ead827. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954078 currentMtime: 1676290954464 2023-02-13 12:22:38,829 INFO [HFileArchiver-9] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-48186611db074ebeab6e0941597e1c9e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-48186611db074ebeab6e0941597e1c9e.1676290958815. 2023-02-13 12:22:38,829 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-84c9c4a754f148fc8187a50ff196c186 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-84c9c4a754f148fc8187a50ff196c186.1676290958815. 2023-02-13 12:22:38,829 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-79d6c194793c4877a70ea1a6905cd846 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-79d6c194793c4877a70ea1a6905cd846.1676290958815. 2023-02-13 12:22:38,830 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-3e370cf8b62343469a87e9049f031efb to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/cf/testRestoreSnapshot-1676290926065=50cc7d0236be777dd672124c8ff93b56-3e370cf8b62343469a87e9049f031efb 2023-02-13 12:22:38,830 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-24b7ea7f7b1d42cfa460732691ead827 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-24b7ea7f7b1d42cfa460732691ead827.1676290958815. 2023-02-13 12:22:38,830 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-d1029a30e4964e55bcbe8195fe1e7315 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/cf/testRestoreSnapshot-1676290926065=6d6fd747f58897a134fa0a460eb9ec99-d1029a30e4964e55bcbe8195fe1e7315 2023-02-13 12:22:38,830 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-79d6c194793c4877a70ea1a6905cd846 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/cf/testRestoreSnapshot-1676290926065=585751c2808ad67a24a61f19e9ac17f5-79d6c194793c4877a70ea1a6905cd846 2023-02-13 12:22:38,830 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-84c9c4a754f148fc8187a50ff196c186 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/cf/testRestoreSnapshot-1676290926065=81a0069ff66a08b05e0e5fe4210e0ce2-84c9c4a754f148fc8187a50ff196c186 2023-02-13 12:22:38,831 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-48186611db074ebeab6e0941597e1c9e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/cf/testRestoreSnapshot-1676290926065=6e9099611b20e9378ce559b4b3a17188-48186611db074ebeab6e0941597e1c9e 2023-02-13 12:22:38,831 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-24b7ea7f7b1d42cfa460732691ead827 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/cf/testRestoreSnapshot-1676290926065=197a58e90a6010ab47efab725a975e4c-24b7ea7f7b1d42cfa460732691ead827 2023-02-13 12:22:38,833 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits/18.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56/recovered.edits/18.seqid 2023-02-13 12:22:38,833 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits/18.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5/recovered.edits/18.seqid 2023-02-13 12:22:38,833 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits/18.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99/recovered.edits/18.seqid 2023-02-13 12:22:38,834 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits/18.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c/recovered.edits/18.seqid 2023-02-13 12:22:38,835 DEBUG [HFileArchiver-10] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/50cc7d0236be777dd672124c8ff93b56 2023-02-13 12:22:38,835 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits/18.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2/recovered.edits/18.seqid 2023-02-13 12:22:38,836 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6d6fd747f58897a134fa0a460eb9ec99 2023-02-13 12:22:38,836 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/585751c2808ad67a24a61f19e9ac17f5 2023-02-13 12:22:38,836 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/197a58e90a6010ab47efab725a975e4c 2023-02-13 12:22:38,836 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/81a0069ff66a08b05e0e5fe4210e0ce2 2023-02-13 12:22:38,836 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits/18.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188/recovered.edits/18.seqid 2023-02-13 12:22:38,837 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testRestoreSnapshot-1676290926065/6e9099611b20e9378ce559b4b3a17188 2023-02-13 12:22:38,837 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676290926065 regions 2023-02-13 12:22:38,838 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081 2023-02-13 12:22:38,838 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf] 2023-02-13 12:22:38,840 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954091 currentMtime: 1676290954380 2023-02-13 12:22:38,841 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a.1676290958839. 2023-02-13 12:22:38,841 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213b0e70a942b0d435c96e6cc12b4c9dd0a 2023-02-13 12:22:38,842 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954092 currentMtime: 1676290954382 2023-02-13 12:22:38,842 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c.1676290958839. 2023-02-13 12:22:38,843 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-a87ff679a2f3e71d9181a67b7542122c20230213cdeee98b490b417d8af6cd4d726d131c 2023-02-13 12:22:38,844 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954093 currentMtime: 1676290954384 2023-02-13 12:22:38,844 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752.1676290958839. 2023-02-13 12:22:38,845 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b202302135f43fa8c53af47258e717ca32cb22752 2023-02-13 12:22:38,846 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954093 currentMtime: 1676290954386 2023-02-13 12:22:38,846 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469.1676290958839. 2023-02-13 12:22:38,847 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c4ca4238a0b923820dcc509a6f75849b20230213f4cb53f1bf704d3d9a3b0e68ae5b5469 2023-02-13 12:22:38,847 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954094 currentMtime: 1676290954387 2023-02-13 12:22:38,848 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c.1676290958839. 2023-02-13 12:22:38,848 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c2023021382d249f2490241859d868c4330d5849c 2023-02-13 12:22:38,849 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954095 currentMtime: 1676290954389 2023-02-13 12:22:38,849 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af.1676290958839. 2023-02-13 12:22:38,850 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-c81e728d9d4c2f636f067f89cc14862c20230213843ff55656204404a2c54dd1c5f6d6af 2023-02-13 12:22:38,850 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954096 currentMtime: 1676290954393 2023-02-13 12:22:38,851 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e.1676290958839. 2023-02-13 12:22:38,851 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e202302137f749b7234b442659098c54233bebc8e 2023-02-13 12:22:38,852 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954097 currentMtime: 1676290954395 2023-02-13 12:22:38,853 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f.1676290958839. 2023-02-13 12:22:38,853 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-d41d8cd98f00b204e9800998ecf8427e20230213dcbf3c62f7e04b3c8bd6ff02d1ab169f 2023-02-13 12:22:38,854 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954097 currentMtime: 1676290954397 2023-02-13 12:22:38,854 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8.1676290958839. 2023-02-13 12:22:38,855 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302131b9f7273927f4fb49b9b44c9990bd3e8 2023-02-13 12:22:38,855 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954098 currentMtime: 1676290954398 2023-02-13 12:22:38,856 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4.1676290958839. 2023-02-13 12:22:38,856 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-e4da3b7fbbce2345d7772b0674a318d5202302135cd1d4457ac24c6f89d8ae5170195aa4 2023-02-13 12:22:38,857 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954099 currentMtime: 1676290954400 2023-02-13 12:22:38,857 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30.1676290958839. 2023-02-13 12:22:38,858 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213519b7acbb90b46e1a468f6f7f8ef8d30 2023-02-13 12:22:38,858 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f. archiveLen: 0 currentLen: 0 archiveMtime: 1676290954100 currentMtime: 1676290954402 2023-02-13 12:22:38,859 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f.1676290958839. 2023-02-13 12:22:38,859 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081/cf/testRestoreSnapshot-1676290926065=c29be10d60d1b4c8e95e9f2610cc0081-eccbc87e4b5ce2fe28308fd9f2a7baf320230213927c8eba088a452aa4bcc1dc722e2c9f 2023-02-13 12:22:38,860 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testRestoreSnapshot-1676290926065/c29be10d60d1b4c8e95e9f2610cc0081 2023-02-13 12:22:38,861 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:38,864 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676290926065 from hbase:meta 2023-02-13 12:22:38,868 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676290926065' descriptor. 2023-02-13 12:22:38,869 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:38,869 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676290926065' from region states. 2023-02-13 12:22:38,869 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290958869"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:38,869 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290958869"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:38,869 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290958869"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:38,869 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290958869"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:38,869 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290958869"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:38,869 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290958869"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:38,871 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 12:22:38,871 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6e9099611b20e9378ce559b4b3a17188, NAME => 'testRestoreSnapshot-1676290926065,,1676290926089.6e9099611b20e9378ce559b4b3a17188.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 50cc7d0236be777dd672124c8ff93b56, NAME => 'testRestoreSnapshot-1676290926065,1,1676290926089.50cc7d0236be777dd672124c8ff93b56.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 6d6fd747f58897a134fa0a460eb9ec99, NAME => 'testRestoreSnapshot-1676290926065,2,1676290926089.6d6fd747f58897a134fa0a460eb9ec99.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 81a0069ff66a08b05e0e5fe4210e0ce2, NAME => 'testRestoreSnapshot-1676290926065,3,1676290926089.81a0069ff66a08b05e0e5fe4210e0ce2.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 197a58e90a6010ab47efab725a975e4c, NAME => 'testRestoreSnapshot-1676290926065,4,1676290926089.197a58e90a6010ab47efab725a975e4c.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 585751c2808ad67a24a61f19e9ac17f5, NAME => 'testRestoreSnapshot-1676290926065,5,1676290926089.585751c2808ad67a24a61f19e9ac17f5.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 12:22:38,871 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676290926065' as deleted. 2023-02-13 12:22:38,871 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676290926065","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676290958871"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:38,873 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676290926065 state from META 2023-02-13 12:22:38,894 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 2023-02-13 12:22:38,895 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676290926065 in 92 msec 2023-02-13 12:22:39,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-13 12:22:39,057 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676290926065, procId: 201 completed 2023-02-13 12:22:39,064 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "emptySnaptb-1676290926065" 2023-02-13 12:22:39,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676290926065 2023-02-13 12:22:39,068 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "snaptb0-1676290926065" 2023-02-13 12:22:39,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676290926065 2023-02-13 12:22:39,072 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "snaptb1-1676290926065" 2023-02-13 12:22:39,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676290926065 2023-02-13 12:22:39,104 INFO [Listener at localhost.localdomain/46179] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=412 (was 361) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x775e0135-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_867678663_15 at /127.0.0.1:60812 [Waiting for operation #503] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x775e0135-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:62541@0x45bdc0c5-SendThread(127.0.0.1:62541) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: HFileArchiver-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x775e0135-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_811437163_15 at /127.0.0.1:40236 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_811437163_15 at /127.0.0.1:52398 [Waiting for operation #495] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:62541@0x45bdc0c5-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x775e0135-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x8f0548d-shared-pool-30 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:62541@0x45bdc0c5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/220654389.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x8f0548d-shared-pool-29 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x8f0548d-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_811437163_15 at /127.0.0.1:40224 [Waiting for operation #504] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_811437163_15 at /127.0.0.1:40240 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x39338c12-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x8f0548d-shared-pool-31 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x775e0135-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_811437163_15 at /127.0.0.1:60796 [Waiting for operation #524] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=776 (was 692) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=329 (was 245) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 171), AvailableMemoryMB=4344 (was 5026) 2023-02-13 12:22:39,120 INFO [Listener at localhost.localdomain/46179] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=412, OpenFileDescriptor=776, MaxFileDescriptor=60000, SystemLoadAverage=329, ProcessCount=169, AvailableMemoryMB=4343 2023-02-13 12:22:39,123 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$4(2102): Client=jenkins//188.40.62.62 create 'testCorruptedSnapshot-1676290959121', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-13 12:22:39,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676290959121 2023-02-13 12:22:39,125 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676290959121 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 12:22:39,126 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676290959121" procId is: 202 2023-02-13 12:22:39,126 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676290959121 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 12:22:39,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 12:22:39,130 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:39,130 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:39,130 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:39,130 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:39,130 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:39,130 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:39,131 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b empty. 2023-02-13 12:22:39,131 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f empty. 2023-02-13 12:22:39,131 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4 empty. 2023-02-13 12:22:39,131 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6 empty. 2023-02-13 12:22:39,131 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e empty. 2023-02-13 12:22:39,131 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655 empty. 2023-02-13 12:22:39,131 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:39,131 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:39,131 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:39,131 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:39,131 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:39,132 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:39,132 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676290959121 regions 2023-02-13 12:22:39,143 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/.tabledesc/.tableinfo.0000000001 2023-02-13 12:22:39,144 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(7671): creating {ENCODED => 87ca394fc8ade3274c52d0e5f9944ed4, NAME => 'testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676290959121', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:39,144 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(7671): creating {ENCODED => d06a08714767f636bc3c65a7d6d9484b, NAME => 'testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676290959121', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:39,149 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(7671): creating {ENCODED => 33e28a5f3e4095f4c6d557938705d655, NAME => 'testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676290959121', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:39,165 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:39,165 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:39,166 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1603): Closing 87ca394fc8ade3274c52d0e5f9944ed4, disabling compactions & flushes 2023-02-13 12:22:39,166 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1603): Closing d06a08714767f636bc3c65a7d6d9484b, disabling compactions & flushes 2023-02-13 12:22:39,166 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:39,166 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:39,166 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:39,166 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:39,166 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. after waiting 0 ms 2023-02-13 12:22:39,166 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. after waiting 0 ms 2023-02-13 12:22:39,166 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:39,166 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:39,166 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:39,166 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:39,166 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1557): Region close journal for 87ca394fc8ade3274c52d0e5f9944ed4: Waiting for close lock at 1676290959166Disabling compacts and flushes for region at 1676290959166Disabling writes for close at 1676290959166Writing region close event to WAL at 1676290959166Closed at 1676290959166 2023-02-13 12:22:39,166 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1557): Region close journal for d06a08714767f636bc3c65a7d6d9484b: Waiting for close lock at 1676290959166Disabling compacts and flushes for region at 1676290959166Disabling writes for close at 1676290959166Writing region close event to WAL at 1676290959166Closed at 1676290959166 2023-02-13 12:22:39,167 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(7671): creating {ENCODED => 7d3bb04107bdd73eeecd83b3f02eaaf6, NAME => 'testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676290959121', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:39,167 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(7671): creating {ENCODED => 196a5aeb92997e39bb79adbb82b8278f, NAME => 'testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676290959121', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:39,171 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:39,172 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1603): Closing 33e28a5f3e4095f4c6d557938705d655, disabling compactions & flushes 2023-02-13 12:22:39,172 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:39,172 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:39,172 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. after waiting 0 ms 2023-02-13 12:22:39,172 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:39,172 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:39,172 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1557): Region close journal for 33e28a5f3e4095f4c6d557938705d655: Waiting for close lock at 1676290959172Disabling compacts and flushes for region at 1676290959172Disabling writes for close at 1676290959172Writing region close event to WAL at 1676290959172Closed at 1676290959172 2023-02-13 12:22:39,172 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(7671): creating {ENCODED => 1ea40c598bfd4d5145115a7df0f84c7e, NAME => 'testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676290959121', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp 2023-02-13 12:22:39,174 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:39,174 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1603): Closing 196a5aeb92997e39bb79adbb82b8278f, disabling compactions & flushes 2023-02-13 12:22:39,174 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:39,174 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:39,174 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. after waiting 0 ms 2023-02-13 12:22:39,174 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:39,174 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:39,174 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-0] regionserver.HRegion(1557): Region close journal for 196a5aeb92997e39bb79adbb82b8278f: Waiting for close lock at 1676290959174Disabling compacts and flushes for region at 1676290959174Disabling writes for close at 1676290959174Writing region close event to WAL at 1676290959174Closed at 1676290959174 2023-02-13 12:22:39,180 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:39,180 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1603): Closing 1ea40c598bfd4d5145115a7df0f84c7e, disabling compactions & flushes 2023-02-13 12:22:39,180 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:39,180 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:39,180 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. after waiting 0 ms 2023-02-13 12:22:39,180 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:39,180 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:39,180 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-2] regionserver.HRegion(1557): Region close journal for 1ea40c598bfd4d5145115a7df0f84c7e: Waiting for close lock at 1676290959180Disabling compacts and flushes for region at 1676290959180Disabling writes for close at 1676290959180Writing region close event to WAL at 1676290959180Closed at 1676290959180 2023-02-13 12:22:39,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 12:22:39,576 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:39,576 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1603): Closing 7d3bb04107bdd73eeecd83b3f02eaaf6, disabling compactions & flushes 2023-02-13 12:22:39,576 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:39,577 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:39,577 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. after waiting 0 ms 2023-02-13 12:22:39,577 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:39,577 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:39,577 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676290959121-pool-1] regionserver.HRegion(1557): Region close journal for 7d3bb04107bdd73eeecd83b3f02eaaf6: Waiting for close lock at 1676290959576Disabling compacts and flushes for region at 1676290959576Disabling writes for close at 1676290959577 (+1 ms)Writing region close event to WAL at 1676290959577Closed at 1676290959577 2023-02-13 12:22:39,581 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676290959121 execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 12:22:39,584 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290959584"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290959584"}]},"ts":"1676290959584"} 2023-02-13 12:22:39,584 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290959584"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290959584"}]},"ts":"1676290959584"} 2023-02-13 12:22:39,585 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290959584"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290959584"}]},"ts":"1676290959584"} 2023-02-13 12:22:39,585 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290959584"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290959584"}]},"ts":"1676290959584"} 2023-02-13 12:22:39,585 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290959584"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290959584"}]},"ts":"1676290959584"} 2023-02-13 12:22:39,585 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290959584"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290959584"}]},"ts":"1676290959584"} 2023-02-13 12:22:39,588 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 12:22:39,589 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676290959121 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 12:22:39,589 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290959589"}]},"ts":"1676290959589"} 2023-02-13 12:22:39,590 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676290959121, state=ENABLING in hbase:meta 2023-02-13 12:22:39,611 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-02-13 12:22:39,613 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 12:22:39,613 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 12:22:39,613 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 12:22:39,613 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 12:22:39,613 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, ASSIGN}] 2023-02-13 12:22:39,615 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, ASSIGN 2023-02-13 12:22:39,615 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, ASSIGN 2023-02-13 12:22:39,616 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, ASSIGN 2023-02-13 12:22:39,616 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, ASSIGN 2023-02-13 12:22:39,616 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=false 2023-02-13 12:22:39,616 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, ASSIGN 2023-02-13 12:22:39,616 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=false 2023-02-13 12:22:39,616 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=false 2023-02-13 12:22:39,616 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=false 2023-02-13 12:22:39,617 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, ASSIGN 2023-02-13 12:22:39,617 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=false 2023-02-13 12:22:39,618 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=false 2023-02-13 12:22:39,767 INFO [jenkins-hbase16:46677] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 12:22:39,773 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=196a5aeb92997e39bb79adbb82b8278f, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:39,773 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=d06a08714767f636bc3c65a7d6d9484b, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:39,773 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=87ca394fc8ade3274c52d0e5f9944ed4, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:39,773 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290959773"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290959773"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290959773"}]},"ts":"1676290959773"} 2023-02-13 12:22:39,773 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290959772"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290959772"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290959772"}]},"ts":"1676290959772"} 2023-02-13 12:22:39,773 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=7d3bb04107bdd73eeecd83b3f02eaaf6, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:39,773 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=33e28a5f3e4095f4c6d557938705d655, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:39,773 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290959773"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290959773"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290959773"}]},"ts":"1676290959773"} 2023-02-13 12:22:39,773 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290959773"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290959773"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290959773"}]},"ts":"1676290959773"} 2023-02-13 12:22:39,773 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290959772"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290959772"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290959772"}]},"ts":"1676290959772"} 2023-02-13 12:22:39,774 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=204, state=RUNNABLE; OpenRegionProcedure 87ca394fc8ade3274c52d0e5f9944ed4, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:39,775 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=203, state=RUNNABLE; OpenRegionProcedure d06a08714767f636bc3c65a7d6d9484b, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:39,776 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=206, state=RUNNABLE; OpenRegionProcedure 7d3bb04107bdd73eeecd83b3f02eaaf6, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:39,776 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=205, state=RUNNABLE; OpenRegionProcedure 33e28a5f3e4095f4c6d557938705d655, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:39,777 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=207, state=RUNNABLE; OpenRegionProcedure 196a5aeb92997e39bb79adbb82b8278f, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:39,777 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=1ea40c598bfd4d5145115a7df0f84c7e, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:39,777 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290959777"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290959777"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290959777"}]},"ts":"1676290959777"} 2023-02-13 12:22:39,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=208, state=RUNNABLE; OpenRegionProcedure 1ea40c598bfd4d5145115a7df0f84c7e, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:39,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 12:22:39,931 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:39,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 87ca394fc8ade3274c52d0e5f9944ed4, NAME => 'testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:22:39,932 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:39,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:39,933 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:39,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:39,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1ea40c598bfd4d5145115a7df0f84c7e, NAME => 'testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:22:39,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:39,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 33e28a5f3e4095f4c6d557938705d655, NAME => 'testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:22:39,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:39,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:39,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:39,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:39,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:39,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:39,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:39,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:39,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:39,935 INFO [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:39,936 INFO [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:39,936 INFO [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:39,937 DEBUG [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf 2023-02-13 12:22:39,937 DEBUG [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf 2023-02-13 12:22:39,937 DEBUG [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf 2023-02-13 12:22:39,937 DEBUG [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf 2023-02-13 12:22:39,937 DEBUG [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf 2023-02-13 12:22:39,937 DEBUG [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf 2023-02-13 12:22:39,937 INFO [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 87ca394fc8ade3274c52d0e5f9944ed4 columnFamilyName cf 2023-02-13 12:22:39,937 INFO [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 33e28a5f3e4095f4c6d557938705d655 columnFamilyName cf 2023-02-13 12:22:39,937 INFO [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1ea40c598bfd4d5145115a7df0f84c7e columnFamilyName cf 2023-02-13 12:22:39,938 INFO [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] regionserver.HStore(310): Store=87ca394fc8ade3274c52d0e5f9944ed4/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:39,938 INFO [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] regionserver.HStore(310): Store=33e28a5f3e4095f4c6d557938705d655/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:39,938 INFO [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] regionserver.HStore(310): Store=1ea40c598bfd4d5145115a7df0f84c7e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:39,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:39,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:39,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:39,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:39,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:39,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:39,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:39,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:39,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:39,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:39,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:39,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:39,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 87ca394fc8ade3274c52d0e5f9944ed4; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64912878, jitterRate=-0.03272274136543274}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:39,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 1ea40c598bfd4d5145115a7df0f84c7e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64173295, jitterRate=-0.04374338686466217}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:39,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 33e28a5f3e4095f4c6d557938705d655; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64896082, jitterRate=-0.032973021268844604}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:39,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 1ea40c598bfd4d5145115a7df0f84c7e: Running coprocessor pre-open hook at 1676290959934Writing region info on filesystem at 1676290959934Initializing all the Stores at 1676290959935 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290959935Cleaning up temporary data from old regions at 1676290959940 (+5 ms)Cleaning up detritus from prior splits at 1676290959941 (+1 ms)Running coprocessor post-open hooks at 1676290959947 (+6 ms)Region opened successfully at 1676290959947 2023-02-13 12:22:39,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 87ca394fc8ade3274c52d0e5f9944ed4: Running coprocessor pre-open hook at 1676290959933Writing region info on filesystem at 1676290959933Initializing all the Stores at 1676290959935 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290959935Cleaning up temporary data from old regions at 1676290959940 (+5 ms)Cleaning up detritus from prior splits at 1676290959941 (+1 ms)Running coprocessor post-open hooks at 1676290959947 (+6 ms)Region opened successfully at 1676290959947 2023-02-13 12:22:39,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 33e28a5f3e4095f4c6d557938705d655: Running coprocessor pre-open hook at 1676290959934Writing region info on filesystem at 1676290959934Initializing all the Stores at 1676290959935 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290959935Cleaning up temporary data from old regions at 1676290959940 (+5 ms)Cleaning up detritus from prior splits at 1676290959941 (+1 ms)Running coprocessor post-open hooks at 1676290959947 (+6 ms)Region opened successfully at 1676290959947 2023-02-13 12:22:39,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655., pid=212, masterSystemTime=1676290959927 2023-02-13 12:22:39,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e., pid=214, masterSystemTime=1676290959926 2023-02-13 12:22:39,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4., pid=209, masterSystemTime=1676290959926 2023-02-13 12:22:39,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:39,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:39,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:39,952 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=87ca394fc8ade3274c52d0e5f9944ed4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:39,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 196a5aeb92997e39bb79adbb82b8278f, NAME => 'testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:22:39,952 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290959951"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290959951"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290959951"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290959951"}]},"ts":"1676290959951"} 2023-02-13 12:22:39,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:39,952 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:39,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:39,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:39,952 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:39,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:39,952 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=1ea40c598bfd4d5145115a7df0f84c7e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:39,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:39,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d06a08714767f636bc3c65a7d6d9484b, NAME => 'testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:22:39,952 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290959952"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290959952"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290959952"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290959952"}]},"ts":"1676290959952"} 2023-02-13 12:22:39,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:39,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:39,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:39,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:39,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:39,953 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:39,953 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:39,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7d3bb04107bdd73eeecd83b3f02eaaf6, NAME => 'testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:22:39,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:39,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:39,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:39,953 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=33e28a5f3e4095f4c6d557938705d655, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:39,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:39,953 INFO [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:39,953 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290959953"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290959953"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290959953"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290959953"}]},"ts":"1676290959953"} 2023-02-13 12:22:39,954 INFO [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:39,954 INFO [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:39,955 DEBUG [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf 2023-02-13 12:22:39,955 DEBUG [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf 2023-02-13 12:22:39,955 INFO [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 196a5aeb92997e39bb79adbb82b8278f columnFamilyName cf 2023-02-13 12:22:39,956 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=204 2023-02-13 12:22:39,956 DEBUG [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf 2023-02-13 12:22:39,956 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=204, state=SUCCESS; OpenRegionProcedure 87ca394fc8ade3274c52d0e5f9944ed4, server=jenkins-hbase16.apache.org,36049,1676290923148 in 180 msec 2023-02-13 12:22:39,956 INFO [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] regionserver.HStore(310): Store=196a5aeb92997e39bb79adbb82b8278f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:39,956 DEBUG [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf 2023-02-13 12:22:39,957 INFO [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7d3bb04107bdd73eeecd83b3f02eaaf6 columnFamilyName cf 2023-02-13 12:22:39,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:39,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:39,958 INFO [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] regionserver.HStore(310): Store=7d3bb04107bdd73eeecd83b3f02eaaf6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:39,957 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, ASSIGN in 343 msec 2023-02-13 12:22:39,957 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=208 2023-02-13 12:22:39,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=205 2023-02-13 12:22:39,958 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=208, state=SUCCESS; OpenRegionProcedure 1ea40c598bfd4d5145115a7df0f84c7e, server=jenkins-hbase16.apache.org,34079,1676290923111 in 175 msec 2023-02-13 12:22:39,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=205, state=SUCCESS; OpenRegionProcedure 33e28a5f3e4095f4c6d557938705d655, server=jenkins-hbase16.apache.org,34139,1676290923061 in 179 msec 2023-02-13 12:22:39,958 DEBUG [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf 2023-02-13 12:22:39,958 DEBUG [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf 2023-02-13 12:22:39,958 INFO [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d06a08714767f636bc3c65a7d6d9484b columnFamilyName cf 2023-02-13 12:22:39,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:39,959 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, ASSIGN in 344 msec 2023-02-13 12:22:39,959 INFO [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] regionserver.HStore(310): Store=d06a08714767f636bc3c65a7d6d9484b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:39,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:39,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:39,960 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, ASSIGN in 345 msec 2023-02-13 12:22:39,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:39,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:39,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:39,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:39,962 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 196a5aeb92997e39bb79adbb82b8278f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72327699, jitterRate=0.07776670157909393}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:39,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 196a5aeb92997e39bb79adbb82b8278f: Running coprocessor pre-open hook at 1676290959952Writing region info on filesystem at 1676290959952Initializing all the Stores at 1676290959953 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290959953Cleaning up temporary data from old regions at 1676290959958 (+5 ms)Cleaning up detritus from prior splits at 1676290959959 (+1 ms)Running coprocessor post-open hooks at 1676290959962 (+3 ms)Region opened successfully at 1676290959962 2023-02-13 12:22:39,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:39,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f., pid=213, masterSystemTime=1676290959926 2023-02-13 12:22:39,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:39,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 7d3bb04107bdd73eeecd83b3f02eaaf6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63624929, jitterRate=-0.051914677023887634}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:39,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 7d3bb04107bdd73eeecd83b3f02eaaf6: Running coprocessor pre-open hook at 1676290959953Writing region info on filesystem at 1676290959953Initializing all the Stores at 1676290959954 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290959954Cleaning up temporary data from old regions at 1676290959960 (+6 ms)Cleaning up detritus from prior splits at 1676290959960Running coprocessor post-open hooks at 1676290959963 (+3 ms)Region opened successfully at 1676290959963 2023-02-13 12:22:39,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 12:22:39,964 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6., pid=211, masterSystemTime=1676290959927 2023-02-13 12:22:39,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:39,964 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:39,964 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened d06a08714767f636bc3c65a7d6d9484b; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70262930, jitterRate=0.0469992458820343}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:39,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for d06a08714767f636bc3c65a7d6d9484b: Running coprocessor pre-open hook at 1676290959953Writing region info on filesystem at 1676290959953Initializing all the Stores at 1676290959954 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290959954Cleaning up temporary data from old regions at 1676290959960 (+6 ms)Cleaning up detritus from prior splits at 1676290959961 (+1 ms)Running coprocessor post-open hooks at 1676290959964 (+3 ms)Region opened successfully at 1676290959964 2023-02-13 12:22:39,964 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=196a5aeb92997e39bb79adbb82b8278f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:39,964 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290959964"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290959964"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290959964"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290959964"}]},"ts":"1676290959964"} 2023-02-13 12:22:39,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b., pid=210, masterSystemTime=1676290959926 2023-02-13 12:22:39,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:39,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:39,965 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=7d3bb04107bdd73eeecd83b3f02eaaf6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:39,965 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290959965"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290959965"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290959965"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290959965"}]},"ts":"1676290959965"} 2023-02-13 12:22:39,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:39,966 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:39,966 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=d06a08714767f636bc3c65a7d6d9484b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:39,966 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290959966"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290959966"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290959966"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290959966"}]},"ts":"1676290959966"} 2023-02-13 12:22:39,967 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=207 2023-02-13 12:22:39,967 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=207, state=SUCCESS; OpenRegionProcedure 196a5aeb92997e39bb79adbb82b8278f, server=jenkins-hbase16.apache.org,36049,1676290923148 in 189 msec 2023-02-13 12:22:39,968 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=206 2023-02-13 12:22:39,968 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, ASSIGN in 354 msec 2023-02-13 12:22:39,968 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=206, state=SUCCESS; OpenRegionProcedure 7d3bb04107bdd73eeecd83b3f02eaaf6, server=jenkins-hbase16.apache.org,34139,1676290923061 in 190 msec 2023-02-13 12:22:39,968 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=203 2023-02-13 12:22:39,968 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=203, state=SUCCESS; OpenRegionProcedure d06a08714767f636bc3c65a7d6d9484b, server=jenkins-hbase16.apache.org,34079,1676290923111 in 192 msec 2023-02-13 12:22:39,969 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, ASSIGN in 355 msec 2023-02-13 12:22:39,970 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=203, resume processing ppid=202 2023-02-13 12:22:39,970 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, ASSIGN in 355 msec 2023-02-13 12:22:39,970 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676290959121 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 12:22:39,971 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290959970"}]},"ts":"1676290959970"} 2023-02-13 12:22:39,972 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676290959121, state=ENABLED in hbase:meta 2023-02-13 12:22:39,999 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676290959121 execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 12:22:40,000 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676290959121 in 876 msec 2023-02-13 12:22:40,316 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 12:22:40,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 12:22:40,632 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676290959121, procId: 202 completed 2023-02-13 12:22:40,635 DEBUG [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676290959121 2023-02-13 12:22:40,636 DEBUG [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:40,640 INFO [Listener at localhost.localdomain/46179] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 12:22:40,661 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676290959121 2023-02-13 12:22:40,662 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testCorruptedSnapshot-1676290959121 2023-02-13 12:22:40,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676290959121 2023-02-13 12:22:40,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-13 12:22:40,665 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290960665"}]},"ts":"1676290960665"} 2023-02-13 12:22:40,666 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676290959121, state=DISABLING in hbase:meta 2023-02-13 12:22:40,676 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676290959121 to state=DISABLING 2023-02-13 12:22:40,677 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, UNASSIGN}] 2023-02-13 12:22:40,679 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, UNASSIGN 2023-02-13 12:22:40,679 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, UNASSIGN 2023-02-13 12:22:40,679 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, UNASSIGN 2023-02-13 12:22:40,679 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, UNASSIGN 2023-02-13 12:22:40,679 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, UNASSIGN 2023-02-13 12:22:40,680 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=33e28a5f3e4095f4c6d557938705d655, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:40,680 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=1ea40c598bfd4d5145115a7df0f84c7e, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:40,680 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=196a5aeb92997e39bb79adbb82b8278f, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:40,680 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=87ca394fc8ade3274c52d0e5f9944ed4, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:40,680 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=7d3bb04107bdd73eeecd83b3f02eaaf6, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:40,680 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290960680"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290960680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290960680"}]},"ts":"1676290960680"} 2023-02-13 12:22:40,680 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290960680"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290960680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290960680"}]},"ts":"1676290960680"} 2023-02-13 12:22:40,680 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290960680"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290960680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290960680"}]},"ts":"1676290960680"} 2023-02-13 12:22:40,680 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290960680"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290960680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290960680"}]},"ts":"1676290960680"} 2023-02-13 12:22:40,680 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290960680"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290960680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290960680"}]},"ts":"1676290960680"} 2023-02-13 12:22:40,682 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=218, state=RUNNABLE; CloseRegionProcedure 33e28a5f3e4095f4c6d557938705d655, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:40,682 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=219, state=RUNNABLE; CloseRegionProcedure 7d3bb04107bdd73eeecd83b3f02eaaf6, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:40,684 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=217, state=RUNNABLE; CloseRegionProcedure 87ca394fc8ade3274c52d0e5f9944ed4, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:40,685 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=220, state=RUNNABLE; CloseRegionProcedure 196a5aeb92997e39bb79adbb82b8278f, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:40,686 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=221, state=RUNNABLE; CloseRegionProcedure 1ea40c598bfd4d5145115a7df0f84c7e, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:40,691 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, UNASSIGN 2023-02-13 12:22:40,691 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=d06a08714767f636bc3c65a7d6d9484b, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:40,692 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290960691"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290960691"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290960691"}]},"ts":"1676290960691"} 2023-02-13 12:22:40,693 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=216, state=RUNNABLE; CloseRegionProcedure d06a08714767f636bc3c65a7d6d9484b, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:40,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:40,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:40,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 7d3bb04107bdd73eeecd83b3f02eaaf6, disabling compactions & flushes 2023-02-13 12:22:40,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing d06a08714767f636bc3c65a7d6d9484b, disabling compactions & flushes 2023-02-13 12:22:40,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:40,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:40,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:40,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:40,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 87ca394fc8ade3274c52d0e5f9944ed4, disabling compactions & flushes 2023-02-13 12:22:40,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. after waiting 0 ms 2023-02-13 12:22:40,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:40,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:40,840 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:40,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. after waiting 0 ms 2023-02-13 12:22:40,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:40,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:40,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. after waiting 0 ms 2023-02-13 12:22:40,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:40,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:22:40,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:22:40,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:22:40,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:40,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for d06a08714767f636bc3c65a7d6d9484b: Waiting for close lock at 1676290960839Running coprocessor pre-close hooks at 1676290960839Disabling compacts and flushes for region at 1676290960839Disabling writes for close at 1676290960840 (+1 ms)Writing region close event to WAL at 1676290960841 (+1 ms)Running coprocessor post-close hooks at 1676290960849 (+8 ms)Closed at 1676290960850 (+1 ms) 2023-02-13 12:22:40,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:40,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:40,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 87ca394fc8ade3274c52d0e5f9944ed4: Waiting for close lock at 1676290960839Running coprocessor pre-close hooks at 1676290960839Disabling compacts and flushes for region at 1676290960839Disabling writes for close at 1676290960840 (+1 ms)Writing region close event to WAL at 1676290960841 (+1 ms)Running coprocessor post-close hooks at 1676290960851 (+10 ms)Closed at 1676290960851 2023-02-13 12:22:40,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 7d3bb04107bdd73eeecd83b3f02eaaf6: Waiting for close lock at 1676290960839Running coprocessor pre-close hooks at 1676290960839Disabling compacts and flushes for region at 1676290960839Disabling writes for close at 1676290960840 (+1 ms)Writing region close event to WAL at 1676290960841 (+1 ms)Running coprocessor post-close hooks at 1676290960852 (+11 ms)Closed at 1676290960852 2023-02-13 12:22:40,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:40,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:40,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 1ea40c598bfd4d5145115a7df0f84c7e, disabling compactions & flushes 2023-02-13 12:22:40,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:40,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:40,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. after waiting 0 ms 2023-02-13 12:22:40,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:40,854 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=d06a08714767f636bc3c65a7d6d9484b, regionState=CLOSED 2023-02-13 12:22:40,854 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290960854"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290960854"}]},"ts":"1676290960854"} 2023-02-13 12:22:40,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:40,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:40,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 33e28a5f3e4095f4c6d557938705d655, disabling compactions & flushes 2023-02-13 12:22:40,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:40,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:40,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. after waiting 0 ms 2023-02-13 12:22:40,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:40,855 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=7d3bb04107bdd73eeecd83b3f02eaaf6, regionState=CLOSED 2023-02-13 12:22:40,855 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290960855"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290960855"}]},"ts":"1676290960855"} 2023-02-13 12:22:40,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:40,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:40,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 196a5aeb92997e39bb79adbb82b8278f, disabling compactions & flushes 2023-02-13 12:22:40,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:40,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:40,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. after waiting 0 ms 2023-02-13 12:22:40,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:40,856 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=87ca394fc8ade3274c52d0e5f9944ed4, regionState=CLOSED 2023-02-13 12:22:40,856 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290960856"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290960856"}]},"ts":"1676290960856"} 2023-02-13 12:22:40,858 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=216 2023-02-13 12:22:40,858 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=216, state=SUCCESS; CloseRegionProcedure d06a08714767f636bc3c65a7d6d9484b, server=jenkins-hbase16.apache.org,34079,1676290923111 in 163 msec 2023-02-13 12:22:40,865 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=219 2023-02-13 12:22:40,865 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=219, state=SUCCESS; CloseRegionProcedure 7d3bb04107bdd73eeecd83b3f02eaaf6, server=jenkins-hbase16.apache.org,34139,1676290923061 in 175 msec 2023-02-13 12:22:40,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:22:40,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:22:40,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 12:22:40,866 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=217 2023-02-13 12:22:40,866 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, UNASSIGN in 181 msec 2023-02-13 12:22:40,866 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=217, state=SUCCESS; CloseRegionProcedure 87ca394fc8ade3274c52d0e5f9944ed4, server=jenkins-hbase16.apache.org,36049,1676290923148 in 173 msec 2023-02-13 12:22:40,867 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:40,867 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 33e28a5f3e4095f4c6d557938705d655: Waiting for close lock at 1676290960854Running coprocessor pre-close hooks at 1676290960854Disabling compacts and flushes for region at 1676290960854Disabling writes for close at 1676290960855 (+1 ms)Writing region close event to WAL at 1676290960855Running coprocessor post-close hooks at 1676290960866 (+11 ms)Closed at 1676290960867 (+1 ms) 2023-02-13 12:22:40,868 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:40,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 196a5aeb92997e39bb79adbb82b8278f: Waiting for close lock at 1676290960855Running coprocessor pre-close hooks at 1676290960855Disabling compacts and flushes for region at 1676290960855Disabling writes for close at 1676290960855Writing region close event to WAL at 1676290960855Running coprocessor post-close hooks at 1676290960867 (+12 ms)Closed at 1676290960868 (+1 ms) 2023-02-13 12:22:40,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:40,869 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, UNASSIGN in 188 msec 2023-02-13 12:22:40,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 1ea40c598bfd4d5145115a7df0f84c7e: Waiting for close lock at 1676290960853Running coprocessor pre-close hooks at 1676290960853Disabling compacts and flushes for region at 1676290960853Disabling writes for close at 1676290960854 (+1 ms)Writing region close event to WAL at 1676290960854Running coprocessor post-close hooks at 1676290960868 (+14 ms)Closed at 1676290960869 (+1 ms) 2023-02-13 12:22:40,870 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, UNASSIGN in 189 msec 2023-02-13 12:22:40,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:40,871 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=33e28a5f3e4095f4c6d557938705d655, regionState=CLOSED 2023-02-13 12:22:40,871 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290960871"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290960871"}]},"ts":"1676290960871"} 2023-02-13 12:22:40,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:40,871 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=196a5aeb92997e39bb79adbb82b8278f, regionState=CLOSED 2023-02-13 12:22:40,871 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290960871"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290960871"}]},"ts":"1676290960871"} 2023-02-13 12:22:40,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:40,872 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=1ea40c598bfd4d5145115a7df0f84c7e, regionState=CLOSED 2023-02-13 12:22:40,872 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290960872"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290960872"}]},"ts":"1676290960872"} 2023-02-13 12:22:40,873 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=218 2023-02-13 12:22:40,873 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=218, state=SUCCESS; CloseRegionProcedure 33e28a5f3e4095f4c6d557938705d655, server=jenkins-hbase16.apache.org,34139,1676290923061 in 190 msec 2023-02-13 12:22:40,874 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=220 2023-02-13 12:22:40,874 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=220, state=SUCCESS; CloseRegionProcedure 196a5aeb92997e39bb79adbb82b8278f, server=jenkins-hbase16.apache.org,36049,1676290923148 in 187 msec 2023-02-13 12:22:40,874 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, UNASSIGN in 196 msec 2023-02-13 12:22:40,874 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=221 2023-02-13 12:22:40,874 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=221, state=SUCCESS; CloseRegionProcedure 1ea40c598bfd4d5145115a7df0f84c7e, server=jenkins-hbase16.apache.org,34079,1676290923111 in 187 msec 2023-02-13 12:22:40,874 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, UNASSIGN in 197 msec 2023-02-13 12:22:40,875 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=221, resume processing ppid=215 2023-02-13 12:22:40,875 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, UNASSIGN in 197 msec 2023-02-13 12:22:40,875 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290960875"}]},"ts":"1676290960875"} 2023-02-13 12:22:40,876 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676290959121, state=DISABLED in hbase:meta 2023-02-13 12:22:40,884 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676290959121 to state=DISABLED 2023-02-13 12:22:40,886 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676290959121 in 223 msec 2023-02-13 12:22:40,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-13 12:22:40,917 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676290959121, procId: 215 completed 2023-02-13 12:22:40,918 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=emptySnaptb-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 } 2023-02-13 12:22:40,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676290960918 (current time:1676290960918). 2023-02-13 12:22:40,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 12:22:40,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(139): Connect 0x21633eb8 to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:40,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a0d8512, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:40,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:40,937 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:50782, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:40,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21633eb8 to 127.0.0.1:62541 2023-02-13 12:22:40,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:40,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 12:22:40,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 } 2023-02-13 12:22:40,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676290959121, type=EXCLUSIVE 2023-02-13 12:22:40,941 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676290959121, type=EXCLUSIVE 2023-02-13 12:22:40,942 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676290959121, type=EXCLUSIVE; timeout=600000, timestamp=1676291560942 2023-02-13 12:22:40,942 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676290959121 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676290959121 2023-02-13 12:22:40,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 } 2023-02-13 12:22:40,942 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 12:22:40,942 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 12:22:40,960 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676290959121 table=testCorruptedSnapshot-1676290959121 type=DISABLED ttl=0 } 2023-02-13 12:22:40,964 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:40,964 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:40,965 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:40,965 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:40,965 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:40,965 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:40,965 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:40,965 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:40,965 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:40,965 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:40,965 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:40,965 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:40,965 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:40,965 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:40,966 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:40,966 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:40,966 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:40,966 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:40,967 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:40,967 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:40,967 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 12:22:41,193 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:22:41,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 } is done 2023-02-13 12:22:41,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 12:22:41,194 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 12:22:41,400 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676290959121 table=testCorruptedSnapshot-1676290959121 type=DISABLED ttl=0 } as finished. 2023-02-13 12:22:41,400 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676290959121 2023-02-13 12:22:41,401 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/emptySnaptb-1676290959121 2023-02-13 12:22:41,432 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/emptySnaptb-1676290959121 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/emptySnaptb-1676290959121 2023-02-13 12:22:41,448 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676290959121 of table testCorruptedSnapshot-1676290959121 completed 2023-02-13 12:22:41,448 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/emptySnaptb-1676290959121 2023-02-13 12:22:41,448 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/emptySnaptb-1676290959121 2023-02-13 12:22:41,448 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676290959121 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676290959121 at 1676290960942Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676290959121 table=testCorruptedSnapshot-1676290959121 type=DISABLED ttl=0 } at 1676290960960 (+18 ms)Consolidate snapshot: emptySnaptb-1676290959121 at 1676290961400 (+440 ms)Loading Region manifests for emptySnaptb-1676290959121 at 1676290961400Writing data manifest for emptySnaptb-1676290959121 at 1676290961409 (+9 ms)Verifying snapshot: emptySnaptb-1676290959121 at 1676290961418 (+9 ms)Snapshot emptySnaptb-1676290959121 of table testCorruptedSnapshot-1676290959121 completed at 1676290961448 (+30 ms) 2023-02-13 12:22:41,450 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676290959121, type=EXCLUSIVE 2023-02-13 12:22:41,451 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676290959121, type=EXCLUSIVE in 510 msec 2023-02-13 12:22:41,695 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:22:41,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 } is done 2023-02-13 12:22:41,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 12:22:41,697 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676290959121 2023-02-13 12:22:41,698 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testCorruptedSnapshot-1676290959121 2023-02-13 12:22:41,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676290959121 2023-02-13 12:22:41,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 12:22:41,703 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676290959121 2023-02-13 12:22:41,703 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290961703"}]},"ts":"1676290961703"} 2023-02-13 12:22:41,705 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676290959121, state=ENABLING in hbase:meta 2023-02-13 12:22:41,767 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 12:22:41,767 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, ASSIGN}] 2023-02-13 12:22:41,770 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, ASSIGN 2023-02-13 12:22:41,770 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, ASSIGN 2023-02-13 12:22:41,770 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, ASSIGN 2023-02-13 12:22:41,770 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, ASSIGN 2023-02-13 12:22:41,770 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:41,770 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:41,771 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=true 2023-02-13 12:22:41,771 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:41,771 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:41,771 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=true 2023-02-13 12:22:41,771 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=true 2023-02-13 12:22:41,771 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=true 2023-02-13 12:22:41,771 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, ASSIGN 2023-02-13 12:22:41,771 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, ASSIGN 2023-02-13 12:22:41,772 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:41,772 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=true 2023-02-13 12:22:41,772 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:41,772 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=true 2023-02-13 12:22:41,921 INFO [jenkins-hbase16:46677] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 12:22:41,921 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=87ca394fc8ade3274c52d0e5f9944ed4, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:41,921 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=d06a08714767f636bc3c65a7d6d9484b, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:41,921 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=33e28a5f3e4095f4c6d557938705d655, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:41,921 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=7d3bb04107bdd73eeecd83b3f02eaaf6, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:41,921 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=196a5aeb92997e39bb79adbb82b8278f, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:41,922 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290961921"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290961921"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290961921"}]},"ts":"1676290961921"} 2023-02-13 12:22:41,922 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290961921"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290961921"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290961921"}]},"ts":"1676290961921"} 2023-02-13 12:22:41,922 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290961921"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290961921"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290961921"}]},"ts":"1676290961921"} 2023-02-13 12:22:41,922 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290961921"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290961921"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290961921"}]},"ts":"1676290961921"} 2023-02-13 12:22:41,922 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290961921"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290961921"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290961921"}]},"ts":"1676290961921"} 2023-02-13 12:22:41,925 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=233, state=RUNNABLE; OpenRegionProcedure 7d3bb04107bdd73eeecd83b3f02eaaf6, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:41,926 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=232, state=RUNNABLE; OpenRegionProcedure 33e28a5f3e4095f4c6d557938705d655, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:41,928 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=230, state=RUNNABLE; OpenRegionProcedure d06a08714767f636bc3c65a7d6d9484b, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:41,929 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=231, state=RUNNABLE; OpenRegionProcedure 87ca394fc8ade3274c52d0e5f9944ed4, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:41,930 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=234, state=RUNNABLE; OpenRegionProcedure 196a5aeb92997e39bb79adbb82b8278f, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:41,930 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=1ea40c598bfd4d5145115a7df0f84c7e, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:41,931 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290961930"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290961930"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290961930"}]},"ts":"1676290961930"} 2023-02-13 12:22:41,932 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=235, state=RUNNABLE; OpenRegionProcedure 1ea40c598bfd4d5145115a7df0f84c7e, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:41,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 12:22:42,084 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:42,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7d3bb04107bdd73eeecd83b3f02eaaf6, NAME => 'testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:22:42,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:42,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:42,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:42,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:42,087 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:42,087 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:42,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1ea40c598bfd4d5145115a7df0f84c7e, NAME => 'testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:22:42,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 196a5aeb92997e39bb79adbb82b8278f, NAME => 'testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:22:42,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:42,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:42,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:42,088 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:42,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:42,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:42,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:42,089 INFO [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:42,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:42,090 INFO [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:42,090 DEBUG [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf 2023-02-13 12:22:42,090 DEBUG [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf 2023-02-13 12:22:42,090 INFO [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:42,090 INFO [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7d3bb04107bdd73eeecd83b3f02eaaf6 columnFamilyName cf 2023-02-13 12:22:42,090 DEBUG [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf 2023-02-13 12:22:42,090 DEBUG [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf 2023-02-13 12:22:42,090 DEBUG [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf 2023-02-13 12:22:42,091 DEBUG [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf 2023-02-13 12:22:42,091 INFO [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 196a5aeb92997e39bb79adbb82b8278f columnFamilyName cf 2023-02-13 12:22:42,091 INFO [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1ea40c598bfd4d5145115a7df0f84c7e columnFamilyName cf 2023-02-13 12:22:42,091 INFO [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] regionserver.HStore(310): Store=7d3bb04107bdd73eeecd83b3f02eaaf6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:42,091 INFO [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] regionserver.HStore(310): Store=196a5aeb92997e39bb79adbb82b8278f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:42,091 INFO [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] regionserver.HStore(310): Store=1ea40c598bfd4d5145115a7df0f84c7e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:42,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:42,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:42,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:42,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:42,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:42,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:42,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:42,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:42,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:42,095 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 7d3bb04107bdd73eeecd83b3f02eaaf6; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75208728, jitterRate=0.12069737911224365}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:42,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 7d3bb04107bdd73eeecd83b3f02eaaf6: Running coprocessor pre-open hook at 1676290962086Writing region info on filesystem at 1676290962086Initializing all the Stores at 1676290962088 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290962088Cleaning up temporary data from old regions at 1676290962093 (+5 ms)Cleaning up detritus from prior splits at 1676290962093Running coprocessor post-open hooks at 1676290962095 (+2 ms)Region opened successfully at 1676290962095 2023-02-13 12:22:42,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 196a5aeb92997e39bb79adbb82b8278f; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59330987, jitterRate=-0.11589939892292023}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:42,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 196a5aeb92997e39bb79adbb82b8278f: Running coprocessor pre-open hook at 1676290962089Writing region info on filesystem at 1676290962089Initializing all the Stores at 1676290962089Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290962089Cleaning up temporary data from old regions at 1676290962093 (+4 ms)Cleaning up detritus from prior splits at 1676290962094 (+1 ms)Running coprocessor post-open hooks at 1676290962096 (+2 ms)Region opened successfully at 1676290962096 2023-02-13 12:22:42,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6., pid=236, masterSystemTime=1676290962078 2023-02-13 12:22:42,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f., pid=240, masterSystemTime=1676290962081 2023-02-13 12:22:42,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 1ea40c598bfd4d5145115a7df0f84c7e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73030697, jitterRate=0.08824218809604645}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:42,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 1ea40c598bfd4d5145115a7df0f84c7e: Running coprocessor pre-open hook at 1676290962089Writing region info on filesystem at 1676290962089Initializing all the Stores at 1676290962090 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290962090Cleaning up temporary data from old regions at 1676290962093 (+3 ms)Cleaning up detritus from prior splits at 1676290962094 (+1 ms)Running coprocessor post-open hooks at 1676290962096 (+2 ms)Region opened successfully at 1676290962096 2023-02-13 12:22:42,097 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e., pid=241, masterSystemTime=1676290962081 2023-02-13 12:22:42,097 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:42,097 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:42,097 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:42,098 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=196a5aeb92997e39bb79adbb82b8278f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:42,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 87ca394fc8ade3274c52d0e5f9944ed4, NAME => 'testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:22:42,098 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290962097"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290962097"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290962097"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290962097"}]},"ts":"1676290962097"} 2023-02-13 12:22:42,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:42,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:42,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:42,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:42,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:42,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:42,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:42,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:42,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:42,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d06a08714767f636bc3c65a7d6d9484b, NAME => 'testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:22:42,098 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=1ea40c598bfd4d5145115a7df0f84c7e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:42,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:42,098 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=7d3bb04107bdd73eeecd83b3f02eaaf6, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:42,099 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290962098"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290962098"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290962098"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290962098"}]},"ts":"1676290962098"} 2023-02-13 12:22:42,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:42,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 33e28a5f3e4095f4c6d557938705d655, NAME => 'testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:22:42,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:42,099 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290962098"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290962098"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290962098"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290962098"}]},"ts":"1676290962098"} 2023-02-13 12:22:42,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:42,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:42,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:42,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:42,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:42,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:42,100 INFO [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:42,100 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=234 2023-02-13 12:22:42,100 INFO [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:42,100 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=234, state=SUCCESS; OpenRegionProcedure 196a5aeb92997e39bb79adbb82b8278f, server=jenkins-hbase16.apache.org,36049,1676290923148 in 169 msec 2023-02-13 12:22:42,101 DEBUG [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf 2023-02-13 12:22:42,101 DEBUG [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf 2023-02-13 12:22:42,101 DEBUG [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf 2023-02-13 12:22:42,101 DEBUG [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf 2023-02-13 12:22:42,101 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, ASSIGN in 333 msec 2023-02-13 12:22:42,101 INFO [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 87ca394fc8ade3274c52d0e5f9944ed4 columnFamilyName cf 2023-02-13 12:22:42,101 INFO [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d06a08714767f636bc3c65a7d6d9484b columnFamilyName cf 2023-02-13 12:22:42,101 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=235 2023-02-13 12:22:42,102 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=235, state=SUCCESS; OpenRegionProcedure 1ea40c598bfd4d5145115a7df0f84c7e, server=jenkins-hbase16.apache.org,34079,1676290923111 in 168 msec 2023-02-13 12:22:42,102 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=233 2023-02-13 12:22:42,102 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=233, state=SUCCESS; OpenRegionProcedure 7d3bb04107bdd73eeecd83b3f02eaaf6, server=jenkins-hbase16.apache.org,34139,1676290923061 in 175 msec 2023-02-13 12:22:42,102 INFO [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] regionserver.HStore(310): Store=d06a08714767f636bc3c65a7d6d9484b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:42,102 INFO [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] regionserver.HStore(310): Store=87ca394fc8ade3274c52d0e5f9944ed4/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:42,102 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, ASSIGN in 335 msec 2023-02-13 12:22:42,102 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, ASSIGN in 334 msec 2023-02-13 12:22:42,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:42,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:42,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:42,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:42,105 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:42,105 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:42,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 87ca394fc8ade3274c52d0e5f9944ed4; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61279065, jitterRate=-0.08687077462673187}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:42,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened d06a08714767f636bc3c65a7d6d9484b; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63008906, jitterRate=-0.06109413504600525}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:42,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 87ca394fc8ade3274c52d0e5f9944ed4: Running coprocessor pre-open hook at 1676290962099Writing region info on filesystem at 1676290962099Initializing all the Stores at 1676290962100 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290962100Cleaning up temporary data from old regions at 1676290962104 (+4 ms)Cleaning up detritus from prior splits at 1676290962104Running coprocessor post-open hooks at 1676290962106 (+2 ms)Region opened successfully at 1676290962106 2023-02-13 12:22:42,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for d06a08714767f636bc3c65a7d6d9484b: Running coprocessor pre-open hook at 1676290962099Writing region info on filesystem at 1676290962099Initializing all the Stores at 1676290962100 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290962100Cleaning up temporary data from old regions at 1676290962104 (+4 ms)Cleaning up detritus from prior splits at 1676290962104Running coprocessor post-open hooks at 1676290962106 (+2 ms)Region opened successfully at 1676290962106 2023-02-13 12:22:42,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4., pid=239, masterSystemTime=1676290962081 2023-02-13 12:22:42,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b., pid=238, masterSystemTime=1676290962081 2023-02-13 12:22:42,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:42,107 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:42,108 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=d06a08714767f636bc3c65a7d6d9484b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:42,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:42,108 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290962108"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290962108"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290962108"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290962108"}]},"ts":"1676290962108"} 2023-02-13 12:22:42,108 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:42,108 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=87ca394fc8ade3274c52d0e5f9944ed4, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:42,108 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290962108"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290962108"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290962108"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290962108"}]},"ts":"1676290962108"} 2023-02-13 12:22:42,110 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=230 2023-02-13 12:22:42,111 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=230, state=SUCCESS; OpenRegionProcedure d06a08714767f636bc3c65a7d6d9484b, server=jenkins-hbase16.apache.org,34079,1676290923111 in 181 msec 2023-02-13 12:22:42,111 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=231 2023-02-13 12:22:42,111 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=231, state=SUCCESS; OpenRegionProcedure 87ca394fc8ade3274c52d0e5f9944ed4, server=jenkins-hbase16.apache.org,36049,1676290923148 in 181 msec 2023-02-13 12:22:42,111 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, ASSIGN in 344 msec 2023-02-13 12:22:42,112 INFO [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:42,112 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, ASSIGN in 344 msec 2023-02-13 12:22:42,112 DEBUG [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf 2023-02-13 12:22:42,112 DEBUG [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf 2023-02-13 12:22:42,113 INFO [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 33e28a5f3e4095f4c6d557938705d655 columnFamilyName cf 2023-02-13 12:22:42,113 INFO [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] regionserver.HStore(310): Store=33e28a5f3e4095f4c6d557938705d655/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:42,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:42,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:42,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:42,116 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 33e28a5f3e4095f4c6d557938705d655; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61316234, jitterRate=-0.08631691336631775}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:42,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 33e28a5f3e4095f4c6d557938705d655: Running coprocessor pre-open hook at 1676290962099Writing region info on filesystem at 1676290962099Initializing all the Stores at 1676290962111 (+12 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290962111Cleaning up temporary data from old regions at 1676290962115 (+4 ms)Cleaning up detritus from prior splits at 1676290962115Running coprocessor post-open hooks at 1676290962116 (+1 ms)Region opened successfully at 1676290962116 2023-02-13 12:22:42,117 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655., pid=237, masterSystemTime=1676290962078 2023-02-13 12:22:42,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:42,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:42,118 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=33e28a5f3e4095f4c6d557938705d655, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:42,118 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290962118"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290962118"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290962118"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290962118"}]},"ts":"1676290962118"} 2023-02-13 12:22:42,120 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=232 2023-02-13 12:22:42,121 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=232, state=SUCCESS; OpenRegionProcedure 33e28a5f3e4095f4c6d557938705d655, server=jenkins-hbase16.apache.org,34139,1676290923061 in 193 msec 2023-02-13 12:22:42,122 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=232, resume processing ppid=229 2023-02-13 12:22:42,122 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, ASSIGN in 354 msec 2023-02-13 12:22:42,122 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290962122"}]},"ts":"1676290962122"} 2023-02-13 12:22:42,123 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676290959121, state=ENABLED in hbase:meta 2023-02-13 12:22:42,131 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676290959121' was successfully enabled. 2023-02-13 12:22:42,132 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676290959121 in 433 msec 2023-02-13 12:22:42,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 12:22:42,455 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676290959121, procId: 229 completed 2023-02-13 12:22:42,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:42,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:42,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:42,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:42,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:42,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:42,473 DEBUG [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676290959121 2023-02-13 12:22:42,473 DEBUG [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:42,473 INFO [Listener at localhost.localdomain/46179] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 12:22:42,483 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 12:22:42,484 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 12:22:42,485 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 12:22:42,486 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 12:22:42,487 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 12:22:42,492 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676290959121 2023-02-13 12:22:42,492 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testCorruptedSnapshot-1676290959121 2023-02-13 12:22:42,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676290959121 2023-02-13 12:22:42,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 12:22:42,496 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290962496"}]},"ts":"1676290962496"} 2023-02-13 12:22:42,497 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676290959121, state=DISABLING in hbase:meta 2023-02-13 12:22:42,506 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676290959121 to state=DISABLING 2023-02-13 12:22:42,507 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, UNASSIGN}] 2023-02-13 12:22:42,507 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, UNASSIGN 2023-02-13 12:22:42,508 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, UNASSIGN 2023-02-13 12:22:42,508 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, UNASSIGN 2023-02-13 12:22:42,508 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, UNASSIGN 2023-02-13 12:22:42,508 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, UNASSIGN 2023-02-13 12:22:42,508 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=87ca394fc8ade3274c52d0e5f9944ed4, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:42,508 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290962508"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290962508"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290962508"}]},"ts":"1676290962508"} 2023-02-13 12:22:42,508 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=d06a08714767f636bc3c65a7d6d9484b, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:42,508 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=196a5aeb92997e39bb79adbb82b8278f, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:42,509 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290962508"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290962508"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290962508"}]},"ts":"1676290962508"} 2023-02-13 12:22:42,509 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290962508"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290962508"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290962508"}]},"ts":"1676290962508"} 2023-02-13 12:22:42,508 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=1ea40c598bfd4d5145115a7df0f84c7e, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:42,508 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=33e28a5f3e4095f4c6d557938705d655, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:42,509 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290962508"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290962508"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290962508"}]},"ts":"1676290962508"} 2023-02-13 12:22:42,509 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290962508"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290962508"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290962508"}]},"ts":"1676290962508"} 2023-02-13 12:22:42,509 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=244, state=RUNNABLE; CloseRegionProcedure 87ca394fc8ade3274c52d0e5f9944ed4, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:42,510 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=243, state=RUNNABLE; CloseRegionProcedure d06a08714767f636bc3c65a7d6d9484b, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:42,510 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=247, state=RUNNABLE; CloseRegionProcedure 196a5aeb92997e39bb79adbb82b8278f, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:42,511 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=248, state=RUNNABLE; CloseRegionProcedure 1ea40c598bfd4d5145115a7df0f84c7e, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:42,511 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=245, state=RUNNABLE; CloseRegionProcedure 33e28a5f3e4095f4c6d557938705d655, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:42,512 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, UNASSIGN 2023-02-13 12:22:42,513 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=7d3bb04107bdd73eeecd83b3f02eaaf6, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:42,513 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290962513"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290962513"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290962513"}]},"ts":"1676290962513"} 2023-02-13 12:22:42,514 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=246, state=RUNNABLE; CloseRegionProcedure 7d3bb04107bdd73eeecd83b3f02eaaf6, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:42,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:42,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:42,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 87ca394fc8ade3274c52d0e5f9944ed4, disabling compactions & flushes 2023-02-13 12:22:42,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 1ea40c598bfd4d5145115a7df0f84c7e, disabling compactions & flushes 2023-02-13 12:22:42,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:42,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:42,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:42,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:42,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. after waiting 0 ms 2023-02-13 12:22:42,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. after waiting 0 ms 2023-02-13 12:22:42,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:42,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:42,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 1ea40c598bfd4d5145115a7df0f84c7e 1/1 column families, dataSize=21.13 KB heapSize=45.67 KB 2023-02-13 12:22:42,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 87ca394fc8ade3274c52d0e5f9944ed4 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-13 12:22:42,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:42,664 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 33e28a5f3e4095f4c6d557938705d655, disabling compactions & flushes 2023-02-13 12:22:42,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:42,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:42,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. after waiting 0 ms 2023-02-13 12:22:42,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:42,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 33e28a5f3e4095f4c6d557938705d655 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-13 12:22:42,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230213aa8fadba9f7a433d8fa102880691b5ef to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/e4da3b7fbbce2345d7772b0674a318d520230213aa8fadba9f7a433d8fa102880691b5ef 2023-02-13 12:22:42,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c2023021362a2f7ac212d40eeb54cbaecf8bea306 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c81e728d9d4c2f636f067f89cc14862c2023021362a2f7ac212d40eeb54cbaecf8bea306 2023-02-13 12:22:42,700 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302137dd2806de42c407abe1d660f6e33bb0e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c4ca4238a0b923820dcc509a6f75849b202302137dd2806de42c407abe1d660f6e33bb0e 2023-02-13 12:22:42,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/.tmp/cf/be23b4ced87a48ac99422333b225266e 2023-02-13 12:22:42,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=21.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/.tmp/cf/ba337c399659480588015762b035cf69 2023-02-13 12:22:42,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/.tmp/cf/be23b4ced87a48ac99422333b225266e as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf/be23b4ced87a48ac99422333b225266e 2023-02-13 12:22:42,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/.tmp/cf/ba337c399659480588015762b035cf69 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf/ba337c399659480588015762b035cf69 2023-02-13 12:22:42,723 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf/be23b4ced87a48ac99422333b225266e, entries=33, sequenceid=8, filesize=10.6 K 2023-02-13 12:22:42,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 33e28a5f3e4095f4c6d557938705d655 in 59ms, sequenceid=8, compaction requested=false 2023-02-13 12:22:42,724 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676290959121' 2023-02-13 12:22:42,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf/ba337c399659480588015762b035cf69, entries=323, sequenceid=8, filesize=60.6 K 2023-02-13 12:22:42,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.13 KB/21637, heapSize ~45.66 KB/46752, currentSize=0 B/0 for 1ea40c598bfd4d5145115a7df0f84c7e in 64ms, sequenceid=8, compaction requested=false 2023-02-13 12:22:42,736 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/.tmp/cf/a833b80ee04a4422a90a84639b6945f7 2023-02-13 12:22:42,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:22:42,739 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:42,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 33e28a5f3e4095f4c6d557938705d655: Waiting for close lock at 1676290962664Running coprocessor pre-close hooks at 1676290962664Disabling compacts and flushes for region at 1676290962664Disabling writes for close at 1676290962665 (+1 ms)Obtaining lock to block concurrent updates at 1676290962665Preparing flush snapshotting stores in 33e28a5f3e4095f4c6d557938705d655 at 1676290962665Finished memstore snapshotting testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676290962665Flushing stores of testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. at 1676290962665Flushing 33e28a5f3e4095f4c6d557938705d655/cf: creating writer at 1676290962666 (+1 ms)Flushing mob file 33e28a5f3e4095f4c6d557938705d655/cf: appending metadata at 1676290962670 (+4 ms)Flushing mob file 33e28a5f3e4095f4c6d557938705d655/cf: closing flushed file at 1676290962670Flushing 33e28a5f3e4095f4c6d557938705d655/cf: appending metadata at 1676290962700 (+30 ms)Flushing 33e28a5f3e4095f4c6d557938705d655/cf: closing flushed file at 1676290962700Flushing 33e28a5f3e4095f4c6d557938705d655/cf: reopening flushed file at 1676290962721 (+21 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 33e28a5f3e4095f4c6d557938705d655 in 59ms, sequenceid=8, compaction requested=false at 1676290962724 (+3 ms)Writing region close event to WAL at 1676290962736 (+12 ms)Running coprocessor post-close hooks at 1676290962739 (+3 ms)Closed at 1676290962739 2023-02-13 12:22:42,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/.tmp/cf/a833b80ee04a4422a90a84639b6945f7 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf/a833b80ee04a4422a90a84639b6945f7 2023-02-13 12:22:42,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:42,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:42,741 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 7d3bb04107bdd73eeecd83b3f02eaaf6, disabling compactions & flushes 2023-02-13 12:22:42,741 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:42,741 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:42,741 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. after waiting 0 ms 2023-02-13 12:22:42,741 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:42,741 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 7d3bb04107bdd73eeecd83b3f02eaaf6 1/1 column families, dataSize=2.68 KB heapSize=6.02 KB 2023-02-13 12:22:42,741 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=33e28a5f3e4095f4c6d557938705d655, regionState=CLOSED 2023-02-13 12:22:42,741 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290962741"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290962741"}]},"ts":"1676290962741"} 2023-02-13 12:22:42,741 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:22:42,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:42,743 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 1ea40c598bfd4d5145115a7df0f84c7e: Waiting for close lock at 1676290962662Running coprocessor pre-close hooks at 1676290962662Disabling compacts and flushes for region at 1676290962662Disabling writes for close at 1676290962662Obtaining lock to block concurrent updates at 1676290962662Preparing flush snapshotting stores in 1ea40c598bfd4d5145115a7df0f84c7e at 1676290962662Finished memstore snapshotting testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e., syncing WAL and waiting on mvcc, flushsize=dataSize=21637, getHeapSize=46752, getOffHeapSize=0, getCellsCount=323 at 1676290962662Flushing stores of testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. at 1676290962663 (+1 ms)Flushing 1ea40c598bfd4d5145115a7df0f84c7e/cf: creating writer at 1676290962664 (+1 ms)Flushing mob file 1ea40c598bfd4d5145115a7df0f84c7e/cf: appending metadata at 1676290962671 (+7 ms)Flushing mob file 1ea40c598bfd4d5145115a7df0f84c7e/cf: closing flushed file at 1676290962671Flushing 1ea40c598bfd4d5145115a7df0f84c7e/cf: appending metadata at 1676290962699 (+28 ms)Flushing 1ea40c598bfd4d5145115a7df0f84c7e/cf: closing flushed file at 1676290962700 (+1 ms)Flushing 1ea40c598bfd4d5145115a7df0f84c7e/cf: reopening flushed file at 1676290962723 (+23 ms)Finished flush of dataSize ~21.13 KB/21637, heapSize ~45.66 KB/46752, currentSize=0 B/0 for 1ea40c598bfd4d5145115a7df0f84c7e in 64ms, sequenceid=8, compaction requested=false at 1676290962726 (+3 ms)Writing region close event to WAL at 1676290962736 (+10 ms)Running coprocessor post-close hooks at 1676290962743 (+7 ms)Closed at 1676290962743 2023-02-13 12:22:42,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:42,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf/a833b80ee04a4422a90a84639b6945f7, entries=31, sequenceid=8, filesize=10.2 K 2023-02-13 12:22:42,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:42,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing d06a08714767f636bc3c65a7d6d9484b, disabling compactions & flushes 2023-02-13 12:22:42,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:42,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:42,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. after waiting 0 ms 2023-02-13 12:22:42,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:42,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing d06a08714767f636bc3c65a7d6d9484b 1/1 column families, dataSize=2.62 KB heapSize=5.88 KB 2023-02-13 12:22:42,745 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 87ca394fc8ade3274c52d0e5f9944ed4 in 83ms, sequenceid=8, compaction requested=false 2023-02-13 12:22:42,747 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=1ea40c598bfd4d5145115a7df0f84c7e, regionState=CLOSED 2023-02-13 12:22:42,747 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290962747"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290962747"}]},"ts":"1676290962747"} 2023-02-13 12:22:42,748 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=245 2023-02-13 12:22:42,748 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=245, state=SUCCESS; CloseRegionProcedure 33e28a5f3e4095f4c6d557938705d655, server=jenkins-hbase16.apache.org,34139,1676290923061 in 232 msec 2023-02-13 12:22:42,749 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, UNASSIGN in 241 msec 2023-02-13 12:22:42,750 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=248 2023-02-13 12:22:42,750 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=248, state=SUCCESS; CloseRegionProcedure 1ea40c598bfd4d5145115a7df0f84c7e, server=jenkins-hbase16.apache.org,34079,1676290923111 in 238 msec 2023-02-13 12:22:42,750 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, UNASSIGN in 243 msec 2023-02-13 12:22:42,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 12:22:42,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:22:42,753 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:42,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 87ca394fc8ade3274c52d0e5f9944ed4: Waiting for close lock at 1676290962662Running coprocessor pre-close hooks at 1676290962662Disabling compacts and flushes for region at 1676290962662Disabling writes for close at 1676290962662Obtaining lock to block concurrent updates at 1676290962662Preparing flush snapshotting stores in 87ca394fc8ade3274c52d0e5f9944ed4 at 1676290962662Finished memstore snapshotting testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676290962662Flushing stores of testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. at 1676290962663 (+1 ms)Flushing 87ca394fc8ade3274c52d0e5f9944ed4/cf: creating writer at 1676290962664 (+1 ms)Flushing mob file 87ca394fc8ade3274c52d0e5f9944ed4/cf: appending metadata at 1676290962674 (+10 ms)Flushing mob file 87ca394fc8ade3274c52d0e5f9944ed4/cf: closing flushed file at 1676290962674Flushing 87ca394fc8ade3274c52d0e5f9944ed4/cf: appending metadata at 1676290962702 (+28 ms)Flushing 87ca394fc8ade3274c52d0e5f9944ed4/cf: closing flushed file at 1676290962702Flushing 87ca394fc8ade3274c52d0e5f9944ed4/cf: reopening flushed file at 1676290962740 (+38 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 87ca394fc8ade3274c52d0e5f9944ed4 in 83ms, sequenceid=8, compaction requested=false at 1676290962745 (+5 ms)Writing region close event to WAL at 1676290962748 (+3 ms)Running coprocessor post-close hooks at 1676290962753 (+5 ms)Closed at 1676290962753 2023-02-13 12:22:42,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:42,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:42,754 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 196a5aeb92997e39bb79adbb82b8278f, disabling compactions & flushes 2023-02-13 12:22:42,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:42,754 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:42,754 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. after waiting 0 ms 2023-02-13 12:22:42,754 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:42,754 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=87ca394fc8ade3274c52d0e5f9944ed4, regionState=CLOSED 2023-02-13 12:22:42,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 196a5aeb92997e39bb79adbb82b8278f 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-13 12:22:42,755 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290962754"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290962754"}]},"ts":"1676290962754"} 2023-02-13 12:22:42,757 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=244 2023-02-13 12:22:42,757 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=244, state=SUCCESS; CloseRegionProcedure 87ca394fc8ade3274c52d0e5f9944ed4, server=jenkins-hbase16.apache.org,36049,1676290923148 in 247 msec 2023-02-13 12:22:42,758 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, UNASSIGN in 250 msec 2023-02-13 12:22:42,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230213dd8deb94441f45888a2262e2b35b15af to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213dd8deb94441f45888a2262e2b35b15af 2023-02-13 12:22:42,766 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302135050d8d1737e453eab439f786604743c to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/d41d8cd98f00b204e9800998ecf8427e202302135050d8d1737e453eab439f786604743c 2023-02-13 12:22:42,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230213d39bd5b6e3754dd1aa4da63953b963fd to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/a87ff679a2f3e71d9181a67b7542122c20230213d39bd5b6e3754dd1aa4da63953b963fd 2023-02-13 12:22:42,776 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/.tmp/cf/d05a3db4a1734b67a4f3021bd89612a9 2023-02-13 12:22:42,777 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/.tmp/cf/56811c66c38e468dbda6587eec083945 2023-02-13 12:22:42,777 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/.tmp/cf/7e219cfe3d574cdcadbedee3cd4135f5 2023-02-13 12:22:42,779 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/.tmp/cf/d05a3db4a1734b67a4f3021bd89612a9 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf/d05a3db4a1734b67a4f3021bd89612a9 2023-02-13 12:22:42,779 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/.tmp/cf/56811c66c38e468dbda6587eec083945 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf/56811c66c38e468dbda6587eec083945 2023-02-13 12:22:42,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/.tmp/cf/7e219cfe3d574cdcadbedee3cd4135f5 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/7e219cfe3d574cdcadbedee3cd4135f5 2023-02-13 12:22:42,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf/d05a3db4a1734b67a4f3021bd89612a9, entries=41, sequenceid=8, filesize=12.0 K 2023-02-13 12:22:42,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.68 KB/2745, heapSize ~6 KB/6144, currentSize=0 B/0 for 7d3bb04107bdd73eeecd83b3f02eaaf6 in 41ms, sequenceid=8, compaction requested=false 2023-02-13 12:22:42,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/7e219cfe3d574cdcadbedee3cd4135f5, entries=32, sequenceid=8, filesize=10.4 K 2023-02-13 12:22:42,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf/56811c66c38e468dbda6587eec083945, entries=40, sequenceid=8, filesize=11.8 K 2023-02-13 12:22:42,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 196a5aeb92997e39bb79adbb82b8278f in 28ms, sequenceid=8, compaction requested=false 2023-02-13 12:22:42,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for d06a08714767f636bc3c65a7d6d9484b in 38ms, sequenceid=8, compaction requested=false 2023-02-13 12:22:42,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:22:42,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:22:42,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:42,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:42,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 7d3bb04107bdd73eeecd83b3f02eaaf6: Waiting for close lock at 1676290962741Running coprocessor pre-close hooks at 1676290962741Disabling compacts and flushes for region at 1676290962741Disabling writes for close at 1676290962741Obtaining lock to block concurrent updates at 1676290962741Preparing flush snapshotting stores in 7d3bb04107bdd73eeecd83b3f02eaaf6 at 1676290962741Finished memstore snapshotting testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6., syncing WAL and waiting on mvcc, flushsize=dataSize=2745, getHeapSize=6144, getOffHeapSize=0, getCellsCount=41 at 1676290962741Flushing stores of testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. at 1676290962741Flushing 7d3bb04107bdd73eeecd83b3f02eaaf6/cf: creating writer at 1676290962742 (+1 ms)Flushing mob file 7d3bb04107bdd73eeecd83b3f02eaaf6/cf: appending metadata at 1676290962752 (+10 ms)Flushing mob file 7d3bb04107bdd73eeecd83b3f02eaaf6/cf: closing flushed file at 1676290962752Flushing 7d3bb04107bdd73eeecd83b3f02eaaf6/cf: appending metadata at 1676290962766 (+14 ms)Flushing 7d3bb04107bdd73eeecd83b3f02eaaf6/cf: closing flushed file at 1676290962766Flushing 7d3bb04107bdd73eeecd83b3f02eaaf6/cf: reopening flushed file at 1676290962779 (+13 ms)Finished flush of dataSize ~2.68 KB/2745, heapSize ~6 KB/6144, currentSize=0 B/0 for 7d3bb04107bdd73eeecd83b3f02eaaf6 in 41ms, sequenceid=8, compaction requested=false at 1676290962782 (+3 ms)Writing region close event to WAL at 1676290962782Running coprocessor post-close hooks at 1676290962785 (+3 ms)Closed at 1676290962785 2023-02-13 12:22:42,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for d06a08714767f636bc3c65a7d6d9484b: Waiting for close lock at 1676290962744Running coprocessor pre-close hooks at 1676290962744Disabling compacts and flushes for region at 1676290962744Disabling writes for close at 1676290962744Obtaining lock to block concurrent updates at 1676290962745 (+1 ms)Preparing flush snapshotting stores in d06a08714767f636bc3c65a7d6d9484b at 1676290962745Finished memstore snapshotting testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b., syncing WAL and waiting on mvcc, flushsize=dataSize=2678, getHeapSize=6000, getOffHeapSize=0, getCellsCount=40 at 1676290962745Flushing stores of testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. at 1676290962745Flushing d06a08714767f636bc3c65a7d6d9484b/cf: creating writer at 1676290962747 (+2 ms)Flushing mob file d06a08714767f636bc3c65a7d6d9484b/cf: appending metadata at 1676290962753 (+6 ms)Flushing mob file d06a08714767f636bc3c65a7d6d9484b/cf: closing flushed file at 1676290962753Flushing d06a08714767f636bc3c65a7d6d9484b/cf: appending metadata at 1676290962766 (+13 ms)Flushing d06a08714767f636bc3c65a7d6d9484b/cf: closing flushed file at 1676290962766Flushing d06a08714767f636bc3c65a7d6d9484b/cf: reopening flushed file at 1676290962780 (+14 ms)Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for d06a08714767f636bc3c65a7d6d9484b in 38ms, sequenceid=8, compaction requested=false at 1676290962783 (+3 ms)Writing region close event to WAL at 1676290962783Running coprocessor post-close hooks at 1676290962785 (+2 ms)Closed at 1676290962785 2023-02-13 12:22:42,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:42,786 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=d06a08714767f636bc3c65a7d6d9484b, regionState=CLOSED 2023-02-13 12:22:42,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:42,786 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290962786"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290962786"}]},"ts":"1676290962786"} 2023-02-13 12:22:42,786 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=7d3bb04107bdd73eeecd83b3f02eaaf6, regionState=CLOSED 2023-02-13 12:22:42,787 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290962786"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290962786"}]},"ts":"1676290962786"} 2023-02-13 12:22:42,789 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=243 2023-02-13 12:22:42,789 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=243, state=SUCCESS; CloseRegionProcedure d06a08714767f636bc3c65a7d6d9484b, server=jenkins-hbase16.apache.org,34079,1676290923111 in 277 msec 2023-02-13 12:22:42,789 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=246 2023-02-13 12:22:42,789 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=246, state=SUCCESS; CloseRegionProcedure 7d3bb04107bdd73eeecd83b3f02eaaf6, server=jenkins-hbase16.apache.org,34139,1676290923061 in 274 msec 2023-02-13 12:22:42,790 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 12:22:42,790 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, UNASSIGN in 282 msec 2023-02-13 12:22:42,790 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, UNASSIGN in 282 msec 2023-02-13 12:22:42,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:42,790 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 196a5aeb92997e39bb79adbb82b8278f: Waiting for close lock at 1676290962754Running coprocessor pre-close hooks at 1676290962754Disabling compacts and flushes for region at 1676290962754Disabling writes for close at 1676290962754Obtaining lock to block concurrent updates at 1676290962755 (+1 ms)Preparing flush snapshotting stores in 196a5aeb92997e39bb79adbb82b8278f at 1676290962755Finished memstore snapshotting testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676290962755Flushing stores of testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. at 1676290962755Flushing 196a5aeb92997e39bb79adbb82b8278f/cf: creating writer at 1676290962755Flushing mob file 196a5aeb92997e39bb79adbb82b8278f/cf: appending metadata at 1676290962761 (+6 ms)Flushing mob file 196a5aeb92997e39bb79adbb82b8278f/cf: closing flushed file at 1676290962761Flushing 196a5aeb92997e39bb79adbb82b8278f/cf: appending metadata at 1676290962771 (+10 ms)Flushing 196a5aeb92997e39bb79adbb82b8278f/cf: closing flushed file at 1676290962771Flushing 196a5aeb92997e39bb79adbb82b8278f/cf: reopening flushed file at 1676290962780 (+9 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 196a5aeb92997e39bb79adbb82b8278f in 28ms, sequenceid=8, compaction requested=false at 1676290962782 (+2 ms)Writing region close event to WAL at 1676290962783 (+1 ms)Running coprocessor post-close hooks at 1676290962790 (+7 ms)Closed at 1676290962790 2023-02-13 12:22:42,791 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:42,792 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=196a5aeb92997e39bb79adbb82b8278f, regionState=CLOSED 2023-02-13 12:22:42,792 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290962792"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290962792"}]},"ts":"1676290962792"} 2023-02-13 12:22:42,793 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=247 2023-02-13 12:22:42,794 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=247, state=SUCCESS; CloseRegionProcedure 196a5aeb92997e39bb79adbb82b8278f, server=jenkins-hbase16.apache.org,36049,1676290923148 in 282 msec 2023-02-13 12:22:42,794 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=247, resume processing ppid=242 2023-02-13 12:22:42,795 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, UNASSIGN in 287 msec 2023-02-13 12:22:42,795 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290962795"}]},"ts":"1676290962795"} 2023-02-13 12:22:42,796 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676290959121, state=DISABLED in hbase:meta 2023-02-13 12:22:42,809 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676290959121 to state=DISABLED 2023-02-13 12:22:42,811 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676290959121 in 318 msec 2023-02-13 12:22:43,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 12:22:43,254 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676290959121, procId: 242 completed 2023-02-13 12:22:43,255 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1580): Client=jenkins//188.40.62.62 snapshot request for:{ ss=snaptb0-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 } 2023-02-13 12:22:43,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676290963255 (current time:1676290963255). 2023-02-13 12:22:43,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 12:22:43,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(139): Connect 0x5d6eb3c8 to 127.0.0.1:62541 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 12:22:43,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65c0c023, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 12:22:43,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 12:22:43,278 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:50790, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 12:22:43,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d6eb3c8 to 127.0.0.1:62541 2023-02-13 12:22:43,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:43,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 12:22:43,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 } 2023-02-13 12:22:43,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676290959121, type=EXCLUSIVE 2023-02-13 12:22:43,283 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676290959121, type=EXCLUSIVE 2023-02-13 12:22:43,284 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676290959121, type=EXCLUSIVE; timeout=600000, timestamp=1676291563284 2023-02-13 12:22:43,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 } 2023-02-13 12:22:43,284 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676290959121 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676290959121 2023-02-13 12:22:43,284 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 12:22:43,284 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 12:22:43,297 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676290959121 table=testCorruptedSnapshot-1676290959121 type=DISABLED ttl=0 } 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:43,299 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 12:22:43,301 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf/56811c66c38e468dbda6587eec083945] hfiles 2023-02-13 12:22:43,301 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf/56811c66c38e468dbda6587eec083945 2023-02-13 12:22:43,301 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf/be23b4ced87a48ac99422333b225266e] hfiles 2023-02-13 12:22:43,301 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf/be23b4ced87a48ac99422333b225266e 2023-02-13 12:22:43,301 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf/a833b80ee04a4422a90a84639b6945f7] hfiles 2023-02-13 12:22:43,301 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/7e219cfe3d574cdcadbedee3cd4135f5] hfiles 2023-02-13 12:22:43,301 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf/a833b80ee04a4422a90a84639b6945f7 2023-02-13 12:22:43,301 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/7e219cfe3d574cdcadbedee3cd4135f5 2023-02-13 12:22:43,301 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf/ba337c399659480588015762b035cf69] hfiles 2023-02-13 12:22:43,301 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf/ba337c399659480588015762b035cf69 2023-02-13 12:22:43,302 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf/d05a3db4a1734b67a4f3021bd89612a9] hfiles 2023-02-13 12:22:43,302 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf/d05a3db4a1734b67a4f3021bd89612a9 2023-02-13 12:22:43,306 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/a87ff679a2f3e71d9181a67b7542122c20230213d39bd5b6e3754dd1aa4da63953b963fd, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c4ca4238a0b923820dcc509a6f75849b202302137dd2806de42c407abe1d660f6e33bb0e, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c81e728d9d4c2f636f067f89cc14862c2023021362a2f7ac212d40eeb54cbaecf8bea306, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/d41d8cd98f00b204e9800998ecf8427e202302135050d8d1737e453eab439f786604743c, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/e4da3b7fbbce2345d7772b0674a318d520230213aa8fadba9f7a433d8fa102880691b5ef, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213dd8deb94441f45888a2262e2b35b15af] hfiles 2023-02-13 12:22:43,306 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/a87ff679a2f3e71d9181a67b7542122c20230213d39bd5b6e3754dd1aa4da63953b963fd 2023-02-13 12:22:43,306 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c4ca4238a0b923820dcc509a6f75849b202302137dd2806de42c407abe1d660f6e33bb0e 2023-02-13 12:22:43,306 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c81e728d9d4c2f636f067f89cc14862c2023021362a2f7ac212d40eeb54cbaecf8bea306 2023-02-13 12:22:43,306 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/d41d8cd98f00b204e9800998ecf8427e202302135050d8d1737e453eab439f786604743c 2023-02-13 12:22:43,306 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/e4da3b7fbbce2345d7772b0674a318d520230213aa8fadba9f7a433d8fa102880691b5ef 2023-02-13 12:22:43,307 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213dd8deb94441f45888a2262e2b35b15af 2023-02-13 12:22:43,318 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676290959121 table=testCorruptedSnapshot-1676290959121 type=DISABLED ttl=0 } as finished. 2023-02-13 12:22:43,318 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676290959121 2023-02-13 12:22:43,318 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/snaptb0-1676290959121 2023-02-13 12:22:43,335 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/snaptb0-1676290959121 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/snaptb0-1676290959121 2023-02-13 12:22:43,354 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676290959121 of table testCorruptedSnapshot-1676290959121 completed 2023-02-13 12:22:43,354 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/snaptb0-1676290959121 2023-02-13 12:22:43,354 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.hbase-snapshot/.tmp/snaptb0-1676290959121 2023-02-13 12:22:43,355 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676290959121 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676290959121 at 1676290963284Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676290959121 table=testCorruptedSnapshot-1676290959121 type=DISABLED ttl=0 } at 1676290963297 (+13 ms)Consolidate snapshot: snaptb0-1676290959121 at 1676290963318 (+21 ms)Loading Region manifests for snaptb0-1676290959121 at 1676290963318Writing data manifest for snaptb0-1676290959121 at 1676290963322 (+4 ms)Verifying snapshot: snaptb0-1676290959121 at 1676290963328 (+6 ms)Snapshot snaptb0-1676290959121 of table testCorruptedSnapshot-1676290959121 completed at 1676290963354 (+26 ms) 2023-02-13 12:22:43,356 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676290959121, type=EXCLUSIVE 2023-02-13 12:22:43,357 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676290959121, type=EXCLUSIVE in 74 msec 2023-02-13 12:22:43,535 DEBUG [Listener at localhost.localdomain/46179] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 12:22:43,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 } is done 2023-02-13 12:22:43,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676290959121 table=testCorruptedSnapshot-1676290959121 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 12:22:43,539 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676290959121 2023-02-13 12:22:43,539 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$10(2372): Client=jenkins//188.40.62.62 enable testCorruptedSnapshot-1676290959121 2023-02-13 12:22:43,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676290959121 2023-02-13 12:22:43,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 12:22:43,545 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676290959121 2023-02-13 12:22:43,546 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290963546"}]},"ts":"1676290963546"} 2023-02-13 12:22:43,547 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676290959121, state=ENABLING in hbase:meta 2023-02-13 12:22:43,557 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 12:22:43,557 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, ASSIGN}] 2023-02-13 12:22:43,557 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, ASSIGN 2023-02-13 12:22:43,558 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, ASSIGN 2023-02-13 12:22:43,558 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, ASSIGN 2023-02-13 12:22:43,558 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, ASSIGN 2023-02-13 12:22:43,558 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, ASSIGN 2023-02-13 12:22:43,558 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:43,558 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:43,558 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:43,558 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=true 2023-02-13 12:22:43,558 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:43,558 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=true 2023-02-13 12:22:43,558 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=true 2023-02-13 12:22:43,558 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:43,558 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,36049,1676290923148; forceNewPlan=false, retain=true 2023-02-13 12:22:43,559 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34139,1676290923061; forceNewPlan=false, retain=true 2023-02-13 12:22:43,559 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, ASSIGN 2023-02-13 12:22:43,559 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:43,559 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,34079,1676290923111; forceNewPlan=false, retain=true 2023-02-13 12:22:43,709 INFO [jenkins-hbase16:46677] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 12:22:43,709 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=33e28a5f3e4095f4c6d557938705d655, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:43,709 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=d06a08714767f636bc3c65a7d6d9484b, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:43,709 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=7d3bb04107bdd73eeecd83b3f02eaaf6, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:43,709 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290963709"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290963709"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290963709"}]},"ts":"1676290963709"} 2023-02-13 12:22:43,709 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=87ca394fc8ade3274c52d0e5f9944ed4, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:43,709 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290963709"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290963709"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290963709"}]},"ts":"1676290963709"} 2023-02-13 12:22:43,709 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=196a5aeb92997e39bb79adbb82b8278f, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:43,710 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290963709"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290963709"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290963709"}]},"ts":"1676290963709"} 2023-02-13 12:22:43,709 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290963709"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290963709"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290963709"}]},"ts":"1676290963709"} 2023-02-13 12:22:43,709 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290963709"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290963709"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290963709"}]},"ts":"1676290963709"} 2023-02-13 12:22:43,712 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=257, state=RUNNABLE; OpenRegionProcedure d06a08714767f636bc3c65a7d6d9484b, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:43,712 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=258, state=RUNNABLE; OpenRegionProcedure 87ca394fc8ade3274c52d0e5f9944ed4, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:43,714 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=261, state=RUNNABLE; OpenRegionProcedure 196a5aeb92997e39bb79adbb82b8278f, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:43,715 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=260, state=RUNNABLE; OpenRegionProcedure 7d3bb04107bdd73eeecd83b3f02eaaf6, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:43,716 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=259, state=RUNNABLE; OpenRegionProcedure 33e28a5f3e4095f4c6d557938705d655, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:43,717 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=1ea40c598bfd4d5145115a7df0f84c7e, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:43,717 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290963717"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290963717"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290963717"}]},"ts":"1676290963717"} 2023-02-13 12:22:43,719 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=262, state=RUNNABLE; OpenRegionProcedure 1ea40c598bfd4d5145115a7df0f84c7e, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:43,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 12:22:43,870 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:43,870 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:43,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d06a08714767f636bc3c65a7d6d9484b, NAME => 'testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 12:22:43,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 87ca394fc8ade3274c52d0e5f9944ed4, NAME => 'testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 12:22:43,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:43,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:43,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:43,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:43,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:43,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:43,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:43,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:43,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 33e28a5f3e4095f4c6d557938705d655, NAME => 'testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 12:22:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:43,874 INFO [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:43,874 INFO [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:43,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:43,875 DEBUG [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf 2023-02-13 12:22:43,875 DEBUG [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf 2023-02-13 12:22:43,875 DEBUG [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf 2023-02-13 12:22:43,875 DEBUG [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf 2023-02-13 12:22:43,875 INFO [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:43,876 INFO [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 87ca394fc8ade3274c52d0e5f9944ed4 columnFamilyName cf 2023-02-13 12:22:43,876 INFO [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d06a08714767f636bc3c65a7d6d9484b columnFamilyName cf 2023-02-13 12:22:43,876 DEBUG [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf 2023-02-13 12:22:43,876 DEBUG [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf 2023-02-13 12:22:43,877 INFO [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 33e28a5f3e4095f4c6d557938705d655 columnFamilyName cf 2023-02-13 12:22:43,882 DEBUG [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf/be23b4ced87a48ac99422333b225266e 2023-02-13 12:22:43,882 DEBUG [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf/56811c66c38e468dbda6587eec083945 2023-02-13 12:22:43,882 INFO [StoreOpener-d06a08714767f636bc3c65a7d6d9484b-1] regionserver.HStore(310): Store=d06a08714767f636bc3c65a7d6d9484b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:43,882 INFO [StoreOpener-33e28a5f3e4095f4c6d557938705d655-1] regionserver.HStore(310): Store=33e28a5f3e4095f4c6d557938705d655/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:43,882 DEBUG [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf/a833b80ee04a4422a90a84639b6945f7 2023-02-13 12:22:43,882 INFO [StoreOpener-87ca394fc8ade3274c52d0e5f9944ed4-1] regionserver.HStore(310): Store=87ca394fc8ade3274c52d0e5f9944ed4/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:43,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:43,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:43,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:43,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:43,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:43,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:43,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:43,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:43,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:43,886 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened d06a08714767f636bc3c65a7d6d9484b; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62073985, jitterRate=-0.0750255435705185}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:43,886 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 87ca394fc8ade3274c52d0e5f9944ed4; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60823733, jitterRate=-0.09365575015544891}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:43,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for d06a08714767f636bc3c65a7d6d9484b: Running coprocessor pre-open hook at 1676290963872Writing region info on filesystem at 1676290963872Initializing all the Stores at 1676290963874 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290963874Cleaning up temporary data from old regions at 1676290963884 (+10 ms)Cleaning up detritus from prior splits at 1676290963884Running coprocessor post-open hooks at 1676290963886 (+2 ms)Region opened successfully at 1676290963886 2023-02-13 12:22:43,886 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 33e28a5f3e4095f4c6d557938705d655; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60175683, jitterRate=-0.10331244766712189}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:43,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 87ca394fc8ade3274c52d0e5f9944ed4: Running coprocessor pre-open hook at 1676290963872Writing region info on filesystem at 1676290963872Initializing all the Stores at 1676290963873 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290963874 (+1 ms)Cleaning up temporary data from old regions at 1676290963884 (+10 ms)Cleaning up detritus from prior splits at 1676290963885 (+1 ms)Running coprocessor post-open hooks at 1676290963887 (+2 ms)Region opened successfully at 1676290963887 2023-02-13 12:22:43,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 33e28a5f3e4095f4c6d557938705d655: Running coprocessor pre-open hook at 1676290963875Writing region info on filesystem at 1676290963875Initializing all the Stores at 1676290963875Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290963875Cleaning up temporary data from old regions at 1676290963884 (+9 ms)Cleaning up detritus from prior splits at 1676290963885 (+1 ms)Running coprocessor post-open hooks at 1676290963887 (+2 ms)Region opened successfully at 1676290963887 2023-02-13 12:22:43,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4., pid=264, masterSystemTime=1676290963865 2023-02-13 12:22:43,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b., pid=263, masterSystemTime=1676290963865 2023-02-13 12:22:43,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655., pid=267, masterSystemTime=1676290963868 2023-02-13 12:22:43,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:43,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:43,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:43,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7d3bb04107bdd73eeecd83b3f02eaaf6, NAME => 'testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 12:22:43,888 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=33e28a5f3e4095f4c6d557938705d655, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:43,889 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290963888"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290963888"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290963888"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290963888"}]},"ts":"1676290963888"} 2023-02-13 12:22:43,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:43,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:43,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:43,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:43,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:43,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:43,889 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=d06a08714767f636bc3c65a7d6d9484b, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:43,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1ea40c598bfd4d5145115a7df0f84c7e, NAME => 'testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 12:22:43,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:43,889 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290963889"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290963889"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290963889"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290963889"}]},"ts":"1676290963889"} 2023-02-13 12:22:43,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:43,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:43,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:43,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:43,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:43,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:43,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:43,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 196a5aeb92997e39bb79adbb82b8278f, NAME => 'testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 12:22:43,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676290959121 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:43,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 12:22:43,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7890): checking encryption for 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:43,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7893): checking classloading for 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:43,891 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=87ca394fc8ade3274c52d0e5f9944ed4, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:43,891 INFO [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:43,891 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290963891"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290963891"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290963891"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290963891"}]},"ts":"1676290963891"} 2023-02-13 12:22:43,891 INFO [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:43,892 INFO [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:43,892 DEBUG [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf 2023-02-13 12:22:43,892 DEBUG [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf 2023-02-13 12:22:43,892 DEBUG [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf 2023-02-13 12:22:43,892 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=259 2023-02-13 12:22:43,892 DEBUG [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf 2023-02-13 12:22:43,892 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=259, state=SUCCESS; OpenRegionProcedure 33e28a5f3e4095f4c6d557938705d655, server=jenkins-hbase16.apache.org,34139,1676290923061 in 174 msec 2023-02-13 12:22:43,893 INFO [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1ea40c598bfd4d5145115a7df0f84c7e columnFamilyName cf 2023-02-13 12:22:43,893 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=257 2023-02-13 12:22:43,893 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=257, state=SUCCESS; OpenRegionProcedure d06a08714767f636bc3c65a7d6d9484b, server=jenkins-hbase16.apache.org,34079,1676290923111 in 179 msec 2023-02-13 12:22:43,893 INFO [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7d3bb04107bdd73eeecd83b3f02eaaf6 columnFamilyName cf 2023-02-13 12:22:43,893 DEBUG [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf 2023-02-13 12:22:43,893 DEBUG [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf 2023-02-13 12:22:43,894 INFO [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 196a5aeb92997e39bb79adbb82b8278f columnFamilyName cf 2023-02-13 12:22:43,895 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, ASSIGN in 335 msec 2023-02-13 12:22:43,895 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=258 2023-02-13 12:22:43,895 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=258, state=SUCCESS; OpenRegionProcedure 87ca394fc8ade3274c52d0e5f9944ed4, server=jenkins-hbase16.apache.org,36049,1676290923148 in 180 msec 2023-02-13 12:22:43,896 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, ASSIGN in 336 msec 2023-02-13 12:22:43,897 DEBUG [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf/ba337c399659480588015762b035cf69 2023-02-13 12:22:43,897 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, ASSIGN in 338 msec 2023-02-13 12:22:43,897 INFO [StoreOpener-1ea40c598bfd4d5145115a7df0f84c7e-1] regionserver.HStore(310): Store=1ea40c598bfd4d5145115a7df0f84c7e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:43,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:43,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:43,899 DEBUG [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/7e219cfe3d574cdcadbedee3cd4135f5 2023-02-13 12:22:43,899 INFO [StoreOpener-196a5aeb92997e39bb79adbb82b8278f-1] regionserver.HStore(310): Store=196a5aeb92997e39bb79adbb82b8278f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:43,898 DEBUG [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf/d05a3db4a1734b67a4f3021bd89612a9 2023-02-13 12:22:43,899 INFO [StoreOpener-7d3bb04107bdd73eeecd83b3f02eaaf6-1] regionserver.HStore(310): Store=7d3bb04107bdd73eeecd83b3f02eaaf6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 12:22:43,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:43,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:43,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:43,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:43,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:43,901 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 1ea40c598bfd4d5145115a7df0f84c7e; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60232205, jitterRate=-0.10247020423412323}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:43,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 1ea40c598bfd4d5145115a7df0f84c7e: Running coprocessor pre-open hook at 1676290963890Writing region info on filesystem at 1676290963890Initializing all the Stores at 1676290963890Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290963890Cleaning up temporary data from old regions at 1676290963899 (+9 ms)Cleaning up detritus from prior splits at 1676290963899Running coprocessor post-open hooks at 1676290963901 (+2 ms)Region opened successfully at 1676290963901 2023-02-13 12:22:43,902 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e., pid=268, masterSystemTime=1676290963865 2023-02-13 12:22:43,902 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:43,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1054): writing seq id for 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:43,903 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 7d3bb04107bdd73eeecd83b3f02eaaf6; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69606603, jitterRate=0.03721921145915985}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:43,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:43,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 7d3bb04107bdd73eeecd83b3f02eaaf6: Running coprocessor pre-open hook at 1676290963889Writing region info on filesystem at 1676290963889Initializing all the Stores at 1676290963890 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290963890Cleaning up temporary data from old regions at 1676290963900 (+10 ms)Cleaning up detritus from prior splits at 1676290963901 (+1 ms)Running coprocessor post-open hooks at 1676290963903 (+2 ms)Region opened successfully at 1676290963903 2023-02-13 12:22:43,903 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:43,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1071): Opened 196a5aeb92997e39bb79adbb82b8278f; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63765140, jitterRate=-0.04982537031173706}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 12:22:43,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(964): Region open journal for 196a5aeb92997e39bb79adbb82b8278f: Running coprocessor pre-open hook at 1676290963890Writing region info on filesystem at 1676290963890Initializing all the Stores at 1676290963891 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676290963891Cleaning up temporary data from old regions at 1676290963901 (+10 ms)Cleaning up detritus from prior splits at 1676290963901Running coprocessor post-open hooks at 1676290963904 (+3 ms)Region opened successfully at 1676290963904 2023-02-13 12:22:43,904 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=1ea40c598bfd4d5145115a7df0f84c7e, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:43,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6., pid=266, masterSystemTime=1676290963868 2023-02-13 12:22:43,904 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290963904"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290963904"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290963904"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290963904"}]},"ts":"1676290963904"} 2023-02-13 12:22:43,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f., pid=265, masterSystemTime=1676290963865 2023-02-13 12:22:43,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:43,905 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:43,905 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=7d3bb04107bdd73eeecd83b3f02eaaf6, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:43,905 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290963905"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290963905"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290963905"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290963905"}]},"ts":"1676290963905"} 2023-02-13 12:22:43,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:43,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:43,906 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=196a5aeb92997e39bb79adbb82b8278f, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:43,906 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290963906"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676290963906"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676290963906"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676290963906"}]},"ts":"1676290963906"} 2023-02-13 12:22:43,907 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=262 2023-02-13 12:22:43,907 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=262, state=SUCCESS; OpenRegionProcedure 1ea40c598bfd4d5145115a7df0f84c7e, server=jenkins-hbase16.apache.org,34079,1676290923111 in 186 msec 2023-02-13 12:22:43,908 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=260 2023-02-13 12:22:43,908 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, ASSIGN in 350 msec 2023-02-13 12:22:43,908 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=260, state=SUCCESS; OpenRegionProcedure 7d3bb04107bdd73eeecd83b3f02eaaf6, server=jenkins-hbase16.apache.org,34139,1676290923061 in 191 msec 2023-02-13 12:22:43,908 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=261 2023-02-13 12:22:43,908 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=261, state=SUCCESS; OpenRegionProcedure 196a5aeb92997e39bb79adbb82b8278f, server=jenkins-hbase16.apache.org,36049,1676290923148 in 193 msec 2023-02-13 12:22:43,908 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, ASSIGN in 351 msec 2023-02-13 12:22:43,909 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=261, resume processing ppid=256 2023-02-13 12:22:43,909 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, ASSIGN in 351 msec 2023-02-13 12:22:43,909 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290963909"}]},"ts":"1676290963909"} 2023-02-13 12:22:43,910 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676290959121, state=ENABLED in hbase:meta 2023-02-13 12:22:43,918 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676290959121' was successfully enabled. 2023-02-13 12:22:43,919 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676290959121 in 378 msec 2023-02-13 12:22:44,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 12:22:44,298 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676290959121, procId: 256 completed 2023-02-13 12:22:44,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:44,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36049] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:44,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:44,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:44,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34139] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:44,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34079] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 12:22:44,314 DEBUG [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676290959121 2023-02-13 12:22:44,314 DEBUG [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:44,314 INFO [Listener at localhost.localdomain/46179] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 12:22:44,449 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 12:22:44,569 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 12:22:44,670 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 12:22:44,714 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45bdc0c5 to 127.0.0.1:62541 2023-02-13 12:22:44,714 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:44,820 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 12:22:44,960 DEBUG [Listener at localhost.localdomain/46179] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 12:22:45,436 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 12:22:46,057 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: 7e219cfe3d574cdcadbedee3cd4135f5 in the real (hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/7e219cfe3d574cdcadbedee3cd4135f5) or archive (hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/7e219cfe3d574cdcadbedee3cd4135f5) directory for the primary table. 2023-02-13 12:22:46,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 7e219cfe3d574cdcadbedee3cd4135f5 in the real (hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/7e219cfe3d574cdcadbedee3cd4135f5) or archive (hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/7e219cfe3d574cdcadbedee3cd4135f5) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2493) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2489) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 12:22:46,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] ipc.CallRunner(144): callId: 378 service: MasterService methodName: RestoreSnapshot size: 116 connection: 188.40.62.62:39376 deadline: 1676291026041, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 7e219cfe3d574cdcadbedee3cd4135f5 in the real (hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/7e219cfe3d574cdcadbedee3cd4135f5) or archive (hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/7e219cfe3d574cdcadbedee3cd4135f5) directory for the primary table. 2023-02-13 12:22:46,081 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676290959121 2023-02-13 12:22:46,082 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$11(2405): Client=jenkins//188.40.62.62 disable testCorruptedSnapshot-1676290959121 2023-02-13 12:22:46,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676290959121 2023-02-13 12:22:46,085 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290966085"}]},"ts":"1676290966085"} 2023-02-13 12:22:46,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 12:22:46,086 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676290959121, state=DISABLING in hbase:meta 2023-02-13 12:22:46,101 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676290959121 to state=DISABLING 2023-02-13 12:22:46,102 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, UNASSIGN}] 2023-02-13 12:22:46,103 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, UNASSIGN 2023-02-13 12:22:46,103 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, UNASSIGN 2023-02-13 12:22:46,103 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, UNASSIGN 2023-02-13 12:22:46,103 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, UNASSIGN 2023-02-13 12:22:46,103 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, UNASSIGN 2023-02-13 12:22:46,104 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=7d3bb04107bdd73eeecd83b3f02eaaf6, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:46,104 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=33e28a5f3e4095f4c6d557938705d655, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:46,104 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290966104"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290966104"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290966104"}]},"ts":"1676290966104"} 2023-02-13 12:22:46,104 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=196a5aeb92997e39bb79adbb82b8278f, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:46,104 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290966104"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290966104"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290966104"}]},"ts":"1676290966104"} 2023-02-13 12:22:46,105 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290966104"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290966104"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290966104"}]},"ts":"1676290966104"} 2023-02-13 12:22:46,104 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=1ea40c598bfd4d5145115a7df0f84c7e, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:46,105 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=87ca394fc8ade3274c52d0e5f9944ed4, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:46,105 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290966104"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290966104"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290966104"}]},"ts":"1676290966104"} 2023-02-13 12:22:46,105 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290966104"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290966104"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290966104"}]},"ts":"1676290966104"} 2023-02-13 12:22:46,106 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=273, state=RUNNABLE; CloseRegionProcedure 33e28a5f3e4095f4c6d557938705d655, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:46,106 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=274, state=RUNNABLE; CloseRegionProcedure 7d3bb04107bdd73eeecd83b3f02eaaf6, server=jenkins-hbase16.apache.org,34139,1676290923061}] 2023-02-13 12:22:46,107 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=275, state=RUNNABLE; CloseRegionProcedure 196a5aeb92997e39bb79adbb82b8278f, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:46,108 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=276, state=RUNNABLE; CloseRegionProcedure 1ea40c598bfd4d5145115a7df0f84c7e, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:46,108 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=272, state=RUNNABLE; CloseRegionProcedure 87ca394fc8ade3274c52d0e5f9944ed4, server=jenkins-hbase16.apache.org,36049,1676290923148}] 2023-02-13 12:22:46,108 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, UNASSIGN 2023-02-13 12:22:46,110 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=d06a08714767f636bc3c65a7d6d9484b, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:46,110 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290966110"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676290966110"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676290966110"}]},"ts":"1676290966110"} 2023-02-13 12:22:46,111 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=271, state=RUNNABLE; CloseRegionProcedure d06a08714767f636bc3c65a7d6d9484b, server=jenkins-hbase16.apache.org,34079,1676290923111}] 2023-02-13 12:22:46,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:46,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 33e28a5f3e4095f4c6d557938705d655, disabling compactions & flushes 2023-02-13 12:22:46,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:46,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:46,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. after waiting 0 ms 2023-02-13 12:22:46,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:46,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:46,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 33e28a5f3e4095f4c6d557938705d655 1/1 column families, dataSize=2.62 KB heapSize=5.88 KB 2023-02-13 12:22:46,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 196a5aeb92997e39bb79adbb82b8278f, disabling compactions & flushes 2023-02-13 12:22:46,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:46,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:46,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. after waiting 0 ms 2023-02-13 12:22:46,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:46,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 196a5aeb92997e39bb79adbb82b8278f 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-13 12:22:46,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:46,261 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing d06a08714767f636bc3c65a7d6d9484b, disabling compactions & flushes 2023-02-13 12:22:46,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:46,262 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:46,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. after waiting 0 ms 2023-02-13 12:22:46,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:46,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing d06a08714767f636bc3c65a7d6d9484b 1/1 column families, dataSize=2.42 KB heapSize=5.45 KB 2023-02-13 12:22:46,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230213ef70b6f09be64923909be5383073f167 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/a87ff679a2f3e71d9181a67b7542122c20230213ef70b6f09be64923909be5383073f167 2023-02-13 12:22:46,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/.tmp/cf/1d694bff5b71444e8575d1ba2ae2713d 2023-02-13 12:22:46,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/.tmp/cf/1d694bff5b71444e8575d1ba2ae2713d as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/1d694bff5b71444e8575d1ba2ae2713d 2023-02-13 12:22:46,299 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/1d694bff5b71444e8575d1ba2ae2713d, entries=32, sequenceid=15, filesize=10.4 K 2023-02-13 12:22:46,300 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 196a5aeb92997e39bb79adbb82b8278f in 39ms, sequenceid=15, compaction requested=false 2023-02-13 12:22:46,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:22:46,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. 2023-02-13 12:22:46,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 196a5aeb92997e39bb79adbb82b8278f: Waiting for close lock at 1676290966260Running coprocessor pre-close hooks at 1676290966260Disabling compacts and flushes for region at 1676290966260Disabling writes for close at 1676290966260Obtaining lock to block concurrent updates at 1676290966261 (+1 ms)Preparing flush snapshotting stores in 196a5aeb92997e39bb79adbb82b8278f at 1676290966261Finished memstore snapshotting testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676290966264 (+3 ms)Flushing stores of testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f. at 1676290966265 (+1 ms)Flushing 196a5aeb92997e39bb79adbb82b8278f/cf: creating writer at 1676290966267 (+2 ms)Flushing mob file 196a5aeb92997e39bb79adbb82b8278f/cf: appending metadata at 1676290966271 (+4 ms)Flushing mob file 196a5aeb92997e39bb79adbb82b8278f/cf: closing flushed file at 1676290966271Flushing 196a5aeb92997e39bb79adbb82b8278f/cf: appending metadata at 1676290966287 (+16 ms)Flushing 196a5aeb92997e39bb79adbb82b8278f/cf: closing flushed file at 1676290966287Flushing 196a5aeb92997e39bb79adbb82b8278f/cf: reopening flushed file at 1676290966296 (+9 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 196a5aeb92997e39bb79adbb82b8278f in 39ms, sequenceid=15, compaction requested=false at 1676290966300 (+4 ms)Writing region close event to WAL at 1676290966301 (+1 ms)Running coprocessor post-close hooks at 1676290966303 (+2 ms)Closed at 1676290966303 2023-02-13 12:22:46,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:46,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:46,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 87ca394fc8ade3274c52d0e5f9944ed4, disabling compactions & flushes 2023-02-13 12:22:46,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:46,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:46,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. after waiting 0 ms 2023-02-13 12:22:46,305 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=196a5aeb92997e39bb79adbb82b8278f, regionState=CLOSED 2023-02-13 12:22:46,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:46,305 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290966305"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290966305"}]},"ts":"1676290966305"} 2023-02-13 12:22:46,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 87ca394fc8ade3274c52d0e5f9944ed4 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-13 12:22:46,308 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=275 2023-02-13 12:22:46,308 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=275, state=SUCCESS; CloseRegionProcedure 196a5aeb92997e39bb79adbb82b8278f, server=jenkins-hbase16.apache.org,36049,1676290923148 in 200 msec 2023-02-13 12:22:46,308 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=196a5aeb92997e39bb79adbb82b8278f, UNASSIGN in 206 msec 2023-02-13 12:22:46,318 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230213276e7258f18240a7b0e4d2550be905da to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c4ca4238a0b923820dcc509a6f75849b20230213276e7258f18240a7b0e4d2550be905da 2023-02-13 12:22:46,325 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/.tmp/cf/889d75f907274b60bfab75ca777d9d90 2023-02-13 12:22:46,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/.tmp/cf/889d75f907274b60bfab75ca777d9d90 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf/889d75f907274b60bfab75ca777d9d90 2023-02-13 12:22:46,333 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf/889d75f907274b60bfab75ca777d9d90, entries=34, sequenceid=15, filesize=10.8 K 2023-02-13 12:22:46,336 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 87ca394fc8ade3274c52d0e5f9944ed4 in 31ms, sequenceid=15, compaction requested=false 2023-02-13 12:22:46,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 12:22:46,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:22:46,342 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. 2023-02-13 12:22:46,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 87ca394fc8ade3274c52d0e5f9944ed4: Waiting for close lock at 1676290966305Running coprocessor pre-close hooks at 1676290966305Disabling compacts and flushes for region at 1676290966305Disabling writes for close at 1676290966305Obtaining lock to block concurrent updates at 1676290966305Preparing flush snapshotting stores in 87ca394fc8ade3274c52d0e5f9944ed4 at 1676290966305Finished memstore snapshotting testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4., syncing WAL and waiting on mvcc, flushsize=dataSize=2276, getHeapSize=5136, getOffHeapSize=0, getCellsCount=34 at 1676290966305Flushing stores of testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4. at 1676290966306 (+1 ms)Flushing 87ca394fc8ade3274c52d0e5f9944ed4/cf: creating writer at 1676290966306Flushing mob file 87ca394fc8ade3274c52d0e5f9944ed4/cf: appending metadata at 1676290966309 (+3 ms)Flushing mob file 87ca394fc8ade3274c52d0e5f9944ed4/cf: closing flushed file at 1676290966309Flushing 87ca394fc8ade3274c52d0e5f9944ed4/cf: appending metadata at 1676290966318 (+9 ms)Flushing 87ca394fc8ade3274c52d0e5f9944ed4/cf: closing flushed file at 1676290966318Flushing 87ca394fc8ade3274c52d0e5f9944ed4/cf: reopening flushed file at 1676290966330 (+12 ms)Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 87ca394fc8ade3274c52d0e5f9944ed4 in 31ms, sequenceid=15, compaction requested=false at 1676290966336 (+6 ms)Writing region close event to WAL at 1676290966337 (+1 ms)Running coprocessor post-close hooks at 1676290966341 (+4 ms)Closed at 1676290966342 (+1 ms) 2023-02-13 12:22:46,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:46,344 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=87ca394fc8ade3274c52d0e5f9944ed4, regionState=CLOSED 2023-02-13 12:22:46,344 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290966344"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290966344"}]},"ts":"1676290966344"} 2023-02-13 12:22:46,347 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=272 2023-02-13 12:22:46,347 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=272, state=SUCCESS; CloseRegionProcedure 87ca394fc8ade3274c52d0e5f9944ed4, server=jenkins-hbase16.apache.org,36049,1676290923148 in 237 msec 2023-02-13 12:22:46,348 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=87ca394fc8ade3274c52d0e5f9944ed4, UNASSIGN in 245 msec 2023-02-13 12:22:46,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230213798f9d116b4c49f3a58570a9f2be9c15 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c81e728d9d4c2f636f067f89cc14862c20230213798f9d116b4c49f3a58570a9f2be9c15 2023-02-13 12:22:46,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230213d994456faff54f8c92bef2f4d5ad8b5b to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/d41d8cd98f00b204e9800998ecf8427e20230213d994456faff54f8c92bef2f4d5ad8b5b 2023-02-13 12:22:46,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/.tmp/cf/0e436f426c83482495871128a47af36b 2023-02-13 12:22:46,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/.tmp/cf/2a024765843f4ee9877aa0049ee09e87 2023-02-13 12:22:46,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/.tmp/cf/2a024765843f4ee9877aa0049ee09e87 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf/2a024765843f4ee9877aa0049ee09e87 2023-02-13 12:22:46,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/.tmp/cf/0e436f426c83482495871128a47af36b as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf/0e436f426c83482495871128a47af36b 2023-02-13 12:22:46,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf/0e436f426c83482495871128a47af36b, entries=40, sequenceid=15, filesize=11.8 K 2023-02-13 12:22:46,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf/2a024765843f4ee9877aa0049ee09e87, entries=37, sequenceid=15, filesize=11.3 K 2023-02-13 12:22:46,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for 33e28a5f3e4095f4c6d557938705d655 in 450ms, sequenceid=15, compaction requested=false 2023-02-13 12:22:46,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for d06a08714767f636bc3c65a7d6d9484b in 445ms, sequenceid=15, compaction requested=false 2023-02-13 12:22:46,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:22:46,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:22:46,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. 2023-02-13 12:22:46,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. 2023-02-13 12:22:46,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 33e28a5f3e4095f4c6d557938705d655: Waiting for close lock at 1676290966259Running coprocessor pre-close hooks at 1676290966259Disabling compacts and flushes for region at 1676290966259Disabling writes for close at 1676290966259Obtaining lock to block concurrent updates at 1676290966260 (+1 ms)Preparing flush snapshotting stores in 33e28a5f3e4095f4c6d557938705d655 at 1676290966260Finished memstore snapshotting testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655., syncing WAL and waiting on mvcc, flushsize=dataSize=2678, getHeapSize=6000, getOffHeapSize=0, getCellsCount=40 at 1676290966260Flushing stores of testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655. at 1676290966261 (+1 ms)Flushing 33e28a5f3e4095f4c6d557938705d655/cf: creating writer at 1676290966264 (+3 ms)Flushing mob file 33e28a5f3e4095f4c6d557938705d655/cf: appending metadata at 1676290966271 (+7 ms)Flushing mob file 33e28a5f3e4095f4c6d557938705d655/cf: closing flushed file at 1676290966271Flushing 33e28a5f3e4095f4c6d557938705d655/cf: appending metadata at 1676290966692 (+421 ms)Flushing 33e28a5f3e4095f4c6d557938705d655/cf: closing flushed file at 1676290966692Flushing 33e28a5f3e4095f4c6d557938705d655/cf: reopening flushed file at 1676290966706 (+14 ms)Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for 33e28a5f3e4095f4c6d557938705d655 in 450ms, sequenceid=15, compaction requested=false at 1676290966709 (+3 ms)Writing region close event to WAL at 1676290966710 (+1 ms)Running coprocessor post-close hooks at 1676290966713 (+3 ms)Closed at 1676290966714 (+1 ms) 2023-02-13 12:22:46,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for d06a08714767f636bc3c65a7d6d9484b: Waiting for close lock at 1676290966261Running coprocessor pre-close hooks at 1676290966261Disabling compacts and flushes for region at 1676290966261Disabling writes for close at 1676290966264 (+3 ms)Obtaining lock to block concurrent updates at 1676290966264Preparing flush snapshotting stores in d06a08714767f636bc3c65a7d6d9484b at 1676290966264Finished memstore snapshotting testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b., syncing WAL and waiting on mvcc, flushsize=dataSize=2477, getHeapSize=5568, getOffHeapSize=0, getCellsCount=37 at 1676290966265 (+1 ms)Flushing stores of testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b. at 1676290966267 (+2 ms)Flushing d06a08714767f636bc3c65a7d6d9484b/cf: creating writer at 1676290966268 (+1 ms)Flushing mob file d06a08714767f636bc3c65a7d6d9484b/cf: appending metadata at 1676290966271 (+3 ms)Flushing mob file d06a08714767f636bc3c65a7d6d9484b/cf: closing flushed file at 1676290966271Flushing d06a08714767f636bc3c65a7d6d9484b/cf: appending metadata at 1676290966692 (+421 ms)Flushing d06a08714767f636bc3c65a7d6d9484b/cf: closing flushed file at 1676290966692Flushing d06a08714767f636bc3c65a7d6d9484b/cf: reopening flushed file at 1676290966706 (+14 ms)Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for d06a08714767f636bc3c65a7d6d9484b in 445ms, sequenceid=15, compaction requested=false at 1676290966709 (+3 ms)Writing region close event to WAL at 1676290966710 (+1 ms)Running coprocessor post-close hooks at 1676290966714 (+4 ms)Closed at 1676290966714 2023-02-13 12:22:46,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:46,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:46,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 1ea40c598bfd4d5145115a7df0f84c7e, disabling compactions & flushes 2023-02-13 12:22:46,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:46,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:46,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. after waiting 0 ms 2023-02-13 12:22:46,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:46,716 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=d06a08714767f636bc3c65a7d6d9484b, regionState=CLOSED 2023-02-13 12:22:46,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 1ea40c598bfd4d5145115a7df0f84c7e 1/1 column families, dataSize=21.33 KB heapSize=46.09 KB 2023-02-13 12:22:46,716 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290966716"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290966716"}]},"ts":"1676290966716"} 2023-02-13 12:22:46,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:46,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:46,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 7d3bb04107bdd73eeecd83b3f02eaaf6, disabling compactions & flushes 2023-02-13 12:22:46,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:46,716 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=33e28a5f3e4095f4c6d557938705d655, regionState=CLOSED 2023-02-13 12:22:46,717 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290966716"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290966716"}]},"ts":"1676290966716"} 2023-02-13 12:22:46,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:46,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. after waiting 0 ms 2023-02-13 12:22:46,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:46,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 7d3bb04107bdd73eeecd83b3f02eaaf6 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-13 12:22:46,721 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=271 2023-02-13 12:22:46,721 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=271, state=SUCCESS; CloseRegionProcedure d06a08714767f636bc3c65a7d6d9484b, server=jenkins-hbase16.apache.org,34079,1676290923111 in 608 msec 2023-02-13 12:22:46,721 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=273 2023-02-13 12:22:46,721 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=273, state=SUCCESS; CloseRegionProcedure 33e28a5f3e4095f4c6d557938705d655, server=jenkins-hbase16.apache.org,34139,1676290923061 in 613 msec 2023-02-13 12:22:46,722 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=d06a08714767f636bc3c65a7d6d9484b, UNASSIGN in 619 msec 2023-02-13 12:22:46,722 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=33e28a5f3e4095f4c6d557938705d655, UNASSIGN in 619 msec 2023-02-13 12:22:46,734 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ebc8092b6e3d4cbbad0134c4febc68c9 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ebc8092b6e3d4cbbad0134c4febc68c9 2023-02-13 12:22:46,734 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230213f4dabd38a11f458fa94fa011c2a94b34 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/e4da3b7fbbce2345d7772b0674a318d520230213f4dabd38a11f458fa94fa011c2a94b34 2023-02-13 12:22:46,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/.tmp/cf/28f2812ad11a4a3fa5ac115ff6760afa 2023-02-13 12:22:46,745 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=21.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/.tmp/cf/0eb7ffe1fa9d4ab4988ef20dfec42e04 2023-02-13 12:22:46,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/.tmp/cf/28f2812ad11a4a3fa5ac115ff6760afa as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf/28f2812ad11a4a3fa5ac115ff6760afa 2023-02-13 12:22:46,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/.tmp/cf/0eb7ffe1fa9d4ab4988ef20dfec42e04 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf/0eb7ffe1fa9d4ab4988ef20dfec42e04 2023-02-13 12:22:46,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf/28f2812ad11a4a3fa5ac115ff6760afa, entries=31, sequenceid=15, filesize=10.2 K 2023-02-13 12:22:46,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf/0eb7ffe1fa9d4ab4988ef20dfec42e04, entries=326, sequenceid=15, filesize=61.1 K 2023-02-13 12:22:46,750 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.33 KB/21838, heapSize ~46.08 KB/47184, currentSize=0 B/0 for 1ea40c598bfd4d5145115a7df0f84c7e in 34ms, sequenceid=15, compaction requested=false 2023-02-13 12:22:46,750 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 7d3bb04107bdd73eeecd83b3f02eaaf6 in 32ms, sequenceid=15, compaction requested=false 2023-02-13 12:22:46,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:22:46,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 12:22:46,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. 2023-02-13 12:22:46,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. 2023-02-13 12:22:46,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 7d3bb04107bdd73eeecd83b3f02eaaf6: Waiting for close lock at 1676290966716Running coprocessor pre-close hooks at 1676290966716Disabling compacts and flushes for region at 1676290966716Disabling writes for close at 1676290966718 (+2 ms)Obtaining lock to block concurrent updates at 1676290966718Preparing flush snapshotting stores in 7d3bb04107bdd73eeecd83b3f02eaaf6 at 1676290966718Finished memstore snapshotting testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676290966718Flushing stores of testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6. at 1676290966718Flushing 7d3bb04107bdd73eeecd83b3f02eaaf6/cf: creating writer at 1676290966719 (+1 ms)Flushing mob file 7d3bb04107bdd73eeecd83b3f02eaaf6/cf: appending metadata at 1676290966725 (+6 ms)Flushing mob file 7d3bb04107bdd73eeecd83b3f02eaaf6/cf: closing flushed file at 1676290966725Flushing 7d3bb04107bdd73eeecd83b3f02eaaf6/cf: appending metadata at 1676290966735 (+10 ms)Flushing 7d3bb04107bdd73eeecd83b3f02eaaf6/cf: closing flushed file at 1676290966735Flushing 7d3bb04107bdd73eeecd83b3f02eaaf6/cf: reopening flushed file at 1676290966747 (+12 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 7d3bb04107bdd73eeecd83b3f02eaaf6 in 32ms, sequenceid=15, compaction requested=false at 1676290966750 (+3 ms)Writing region close event to WAL at 1676290966751 (+1 ms)Running coprocessor post-close hooks at 1676290966753 (+2 ms)Closed at 1676290966755 (+2 ms) 2023-02-13 12:22:46,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 1ea40c598bfd4d5145115a7df0f84c7e: Waiting for close lock at 1676290966716Running coprocessor pre-close hooks at 1676290966716Disabling compacts and flushes for region at 1676290966716Disabling writes for close at 1676290966716Obtaining lock to block concurrent updates at 1676290966716Preparing flush snapshotting stores in 1ea40c598bfd4d5145115a7df0f84c7e at 1676290966716Finished memstore snapshotting testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e., syncing WAL and waiting on mvcc, flushsize=dataSize=21838, getHeapSize=47184, getOffHeapSize=0, getCellsCount=326 at 1676290966716Flushing stores of testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e. at 1676290966717 (+1 ms)Flushing 1ea40c598bfd4d5145115a7df0f84c7e/cf: creating writer at 1676290966718 (+1 ms)Flushing mob file 1ea40c598bfd4d5145115a7df0f84c7e/cf: appending metadata at 1676290966727 (+9 ms)Flushing mob file 1ea40c598bfd4d5145115a7df0f84c7e/cf: closing flushed file at 1676290966727Flushing 1ea40c598bfd4d5145115a7df0f84c7e/cf: appending metadata at 1676290966735 (+8 ms)Flushing 1ea40c598bfd4d5145115a7df0f84c7e/cf: closing flushed file at 1676290966735Flushing 1ea40c598bfd4d5145115a7df0f84c7e/cf: reopening flushed file at 1676290966747 (+12 ms)Finished flush of dataSize ~21.33 KB/21838, heapSize ~46.08 KB/47184, currentSize=0 B/0 for 1ea40c598bfd4d5145115a7df0f84c7e in 34ms, sequenceid=15, compaction requested=false at 1676290966750 (+3 ms)Writing region close event to WAL at 1676290966751 (+1 ms)Running coprocessor post-close hooks at 1676290966753 (+2 ms)Closed at 1676290966755 (+2 ms) 2023-02-13 12:22:46,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:46,756 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=7d3bb04107bdd73eeecd83b3f02eaaf6, regionState=CLOSED 2023-02-13 12:22:46,756 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676290966756"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290966756"}]},"ts":"1676290966756"} 2023-02-13 12:22:46,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:46,757 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=1ea40c598bfd4d5145115a7df0f84c7e, regionState=CLOSED 2023-02-13 12:22:46,757 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676290966757"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676290966757"}]},"ts":"1676290966757"} 2023-02-13 12:22:46,758 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=274 2023-02-13 12:22:46,758 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=274, state=SUCCESS; CloseRegionProcedure 7d3bb04107bdd73eeecd83b3f02eaaf6, server=jenkins-hbase16.apache.org,34139,1676290923061 in 651 msec 2023-02-13 12:22:46,759 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=276 2023-02-13 12:22:46,759 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=276, state=SUCCESS; CloseRegionProcedure 1ea40c598bfd4d5145115a7df0f84c7e, server=jenkins-hbase16.apache.org,34079,1676290923111 in 650 msec 2023-02-13 12:22:46,759 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=7d3bb04107bdd73eeecd83b3f02eaaf6, UNASSIGN in 656 msec 2023-02-13 12:22:46,760 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=276, resume processing ppid=270 2023-02-13 12:22:46,760 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676290959121, region=1ea40c598bfd4d5145115a7df0f84c7e, UNASSIGN in 657 msec 2023-02-13 12:22:46,761 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676290966761"}]},"ts":"1676290966761"} 2023-02-13 12:22:46,762 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676290959121, state=DISABLED in hbase:meta 2023-02-13 12:22:46,772 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676290959121 to state=DISABLED 2023-02-13 12:22:46,774 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676290959121 in 691 msec 2023-02-13 12:22:46,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 12:22:46,840 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676290959121, procId: 270 completed 2023-02-13 12:22:46,842 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.HMaster$5(2215): Client=jenkins//188.40.62.62 delete testCorruptedSnapshot-1676290959121 2023-02-13 12:22:46,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676290959121 2023-02-13 12:22:46,847 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676290959121 2023-02-13 12:22:46,849 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676290959121 2023-02-13 12:22:46,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-13 12:22:46,861 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:46,861 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:46,861 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:46,861 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:46,861 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:46,861 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:46,862 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/recovered.edits] 2023-02-13 12:22:46,863 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/recovered.edits] 2023-02-13 12:22:46,863 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/recovered.edits] 2023-02-13 12:22:46,863 DEBUG [HFileArchiver-10] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/recovered.edits] 2023-02-13 12:22:46,863 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/recovered.edits] 2023-02-13 12:22:46,863 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf, FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/recovered.edits] 2023-02-13 12:22:46,867 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf/0e436f426c83482495871128a47af36b to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf/0e436f426c83482495871128a47af36b 2023-02-13 12:22:46,867 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf/28f2812ad11a4a3fa5ac115ff6760afa to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf/28f2812ad11a4a3fa5ac115ff6760afa 2023-02-13 12:22:46,867 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf/0eb7ffe1fa9d4ab4988ef20dfec42e04 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf/0eb7ffe1fa9d4ab4988ef20dfec42e04 2023-02-13 12:22:46,867 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf/2a024765843f4ee9877aa0049ee09e87 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf/2a024765843f4ee9877aa0049ee09e87 2023-02-13 12:22:46,868 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/1d694bff5b71444e8575d1ba2ae2713d to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/cf/1d694bff5b71444e8575d1ba2ae2713d 2023-02-13 12:22:46,868 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf/889d75f907274b60bfab75ca777d9d90 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf/889d75f907274b60bfab75ca777d9d90 2023-02-13 12:22:46,868 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf/be23b4ced87a48ac99422333b225266e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/cf/be23b4ced87a48ac99422333b225266e 2023-02-13 12:22:46,869 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf/d05a3db4a1734b67a4f3021bd89612a9 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/cf/d05a3db4a1734b67a4f3021bd89612a9 2023-02-13 12:22:46,869 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf/ba337c399659480588015762b035cf69 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/cf/ba337c399659480588015762b035cf69 2023-02-13 12:22:46,869 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf/56811c66c38e468dbda6587eec083945 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/cf/56811c66c38e468dbda6587eec083945 2023-02-13 12:22:46,869 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf/a833b80ee04a4422a90a84639b6945f7 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/cf/a833b80ee04a4422a90a84639b6945f7 2023-02-13 12:22:46,871 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/recovered.edits/18.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f/recovered.edits/18.seqid 2023-02-13 12:22:46,872 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/recovered.edits/18.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655/recovered.edits/18.seqid 2023-02-13 12:22:46,872 DEBUG [HFileArchiver-10] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/196a5aeb92997e39bb79adbb82b8278f 2023-02-13 12:22:46,872 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/33e28a5f3e4095f4c6d557938705d655 2023-02-13 12:22:46,872 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/recovered.edits/18.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4/recovered.edits/18.seqid 2023-02-13 12:22:46,873 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/recovered.edits/18.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6/recovered.edits/18.seqid 2023-02-13 12:22:46,873 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/recovered.edits/18.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e/recovered.edits/18.seqid 2023-02-13 12:22:46,873 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/87ca394fc8ade3274c52d0e5f9944ed4 2023-02-13 12:22:46,873 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/recovered.edits/18.seqid to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b/recovered.edits/18.seqid 2023-02-13 12:22:46,873 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/1ea40c598bfd4d5145115a7df0f84c7e 2023-02-13 12:22:46,873 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/7d3bb04107bdd73eeecd83b3f02eaaf6 2023-02-13 12:22:46,874 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/.tmp/data/default/testCorruptedSnapshot-1676290959121/d06a08714767f636bc3c65a7d6d9484b 2023-02-13 12:22:46,874 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676290959121 regions 2023-02-13 12:22:46,874 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a 2023-02-13 12:22:46,874 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf] 2023-02-13 12:22:46,884 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/a87ff679a2f3e71d9181a67b7542122c20230213d39bd5b6e3754dd1aa4da63953b963fd to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/a87ff679a2f3e71d9181a67b7542122c20230213d39bd5b6e3754dd1aa4da63953b963fd 2023-02-13 12:22:46,885 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/a87ff679a2f3e71d9181a67b7542122c20230213ef70b6f09be64923909be5383073f167 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/a87ff679a2f3e71d9181a67b7542122c20230213ef70b6f09be64923909be5383073f167 2023-02-13 12:22:46,886 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c4ca4238a0b923820dcc509a6f75849b20230213276e7258f18240a7b0e4d2550be905da to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c4ca4238a0b923820dcc509a6f75849b20230213276e7258f18240a7b0e4d2550be905da 2023-02-13 12:22:46,887 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c4ca4238a0b923820dcc509a6f75849b202302137dd2806de42c407abe1d660f6e33bb0e to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c4ca4238a0b923820dcc509a6f75849b202302137dd2806de42c407abe1d660f6e33bb0e 2023-02-13 12:22:46,887 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c81e728d9d4c2f636f067f89cc14862c2023021362a2f7ac212d40eeb54cbaecf8bea306 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c81e728d9d4c2f636f067f89cc14862c2023021362a2f7ac212d40eeb54cbaecf8bea306 2023-02-13 12:22:46,888 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c81e728d9d4c2f636f067f89cc14862c20230213798f9d116b4c49f3a58570a9f2be9c15 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/c81e728d9d4c2f636f067f89cc14862c20230213798f9d116b4c49f3a58570a9f2be9c15 2023-02-13 12:22:46,889 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/d41d8cd98f00b204e9800998ecf8427e202302135050d8d1737e453eab439f786604743c to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/d41d8cd98f00b204e9800998ecf8427e202302135050d8d1737e453eab439f786604743c 2023-02-13 12:22:46,889 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/d41d8cd98f00b204e9800998ecf8427e20230213d994456faff54f8c92bef2f4d5ad8b5b to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/d41d8cd98f00b204e9800998ecf8427e20230213d994456faff54f8c92bef2f4d5ad8b5b 2023-02-13 12:22:46,890 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/e4da3b7fbbce2345d7772b0674a318d520230213aa8fadba9f7a433d8fa102880691b5ef to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/e4da3b7fbbce2345d7772b0674a318d520230213aa8fadba9f7a433d8fa102880691b5ef 2023-02-13 12:22:46,891 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/e4da3b7fbbce2345d7772b0674a318d520230213f4dabd38a11f458fa94fa011c2a94b34 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/e4da3b7fbbce2345d7772b0674a318d520230213f4dabd38a11f458fa94fa011c2a94b34 2023-02-13 12:22:46,892 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213dd8deb94441f45888a2262e2b35b15af to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213dd8deb94441f45888a2262e2b35b15af 2023-02-13 12:22:46,893 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ebc8092b6e3d4cbbad0134c4febc68c9 to hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/archive/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213ebc8092b6e3d4cbbad0134c4febc68c9 2023-02-13 12:22:46,893 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/mobdir/data/default/testCorruptedSnapshot-1676290959121/63f9092223d55dc70349fedd28e9f55a 2023-02-13 12:22:46,895 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676290959121 2023-02-13 12:22:46,900 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676290959121 from hbase:meta 2023-02-13 12:22:46,902 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676290959121' descriptor. 2023-02-13 12:22:46,903 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676290959121 2023-02-13 12:22:46,903 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676290959121' from region states. 2023-02-13 12:22:46,903 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290966903"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:46,904 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290966903"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:46,904 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290966903"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:46,904 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290966903"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:46,904 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290966903"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:46,904 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676290966903"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:46,909 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 12:22:46,909 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d06a08714767f636bc3c65a7d6d9484b, NAME => 'testCorruptedSnapshot-1676290959121,,1676290959123.d06a08714767f636bc3c65a7d6d9484b.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 87ca394fc8ade3274c52d0e5f9944ed4, NAME => 'testCorruptedSnapshot-1676290959121,1,1676290959123.87ca394fc8ade3274c52d0e5f9944ed4.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 33e28a5f3e4095f4c6d557938705d655, NAME => 'testCorruptedSnapshot-1676290959121,2,1676290959123.33e28a5f3e4095f4c6d557938705d655.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 7d3bb04107bdd73eeecd83b3f02eaaf6, NAME => 'testCorruptedSnapshot-1676290959121,3,1676290959123.7d3bb04107bdd73eeecd83b3f02eaaf6.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 196a5aeb92997e39bb79adbb82b8278f, NAME => 'testCorruptedSnapshot-1676290959121,4,1676290959123.196a5aeb92997e39bb79adbb82b8278f.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 1ea40c598bfd4d5145115a7df0f84c7e, NAME => 'testCorruptedSnapshot-1676290959121,5,1676290959123.1ea40c598bfd4d5145115a7df0f84c7e.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 12:22:46,909 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676290959121' as deleted. 2023-02-13 12:22:46,909 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676290959121","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676290966909"}]},"ts":"9223372036854775807"} 2023-02-13 12:22:46,911 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676290959121 state from META 2023-02-13 12:22:46,918 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676290959121 2023-02-13 12:22:46,919 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676290959121 in 76 msec 2023-02-13 12:22:47,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-13 12:22:47,101 INFO [Listener at localhost.localdomain/46179] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676290959121, procId: 283 completed 2023-02-13 12:22:47,106 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "emptySnaptb-1676290959121" 2023-02-13 12:22:47,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676290959121 2023-02-13 12:22:47,109 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] master.MasterRpcServices(748): Client=jenkins//188.40.62.62 delete name: "snaptb0-1676290959121" 2023-02-13 12:22:47,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46677] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676290959121 2023-02-13 12:22:47,127 INFO [Listener at localhost.localdomain/46179] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=420 (was 412) Potentially hanging thread: hconnection-0x8f0548d-shared-pool-39 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_480818368_15 at /127.0.0.1:38482 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_480818368_15 at /127.0.0.1:60812 [Waiting for operation #753] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_811437163_15 at /127.0.0.1:38206 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x39338c12-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x8f0548d-shared-pool-38 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_811437163_15 at /127.0.0.1:40236 [Waiting for operation #288] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_480818368_15 at /127.0.0.1:38514 [Waiting for operation #260] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x8f0548d-shared-pool-37 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_811437163_15 at /127.0.0.1:52398 [Waiting for operation #505] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1825703767_15 at /127.0.0.1:55266 [Waiting for operation #13] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1337383090_15 at /127.0.0.1:38494 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x39338c12-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x39338c12-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_480818368_15 at /127.0.0.1:38496 [Waiting for operation #261] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_811437163_15 at /127.0.0.1:40224 [Waiting for operation #513] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2001749296_15 at /127.0.0.1:40240 [Waiting for operation #286] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x39338c12-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1825703767_15 at /127.0.0.1:60796 [Waiting for operation #795] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x39338c12-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x8f0548d-shared-pool-36 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=802 (was 776) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=326 (was 329), ProcessCount=169 (was 169), AvailableMemoryMB=4316 (was 4343) 2023-02-13 12:22:47,128 INFO [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-13 12:22:47,128 INFO [Listener at localhost.localdomain/46179] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-13 12:22:47,129 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20c4cf78 to 127.0.0.1:62541 2023-02-13 12:22:47,129 DEBUG [Listener at localhost.localdomain/46179] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:47,129 DEBUG [Listener at localhost.localdomain/46179] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 12:22:47,129 DEBUG [Listener at localhost.localdomain/46179] util.JVMClusterUtil(257): Found active master hash=697335151, stopped=false 2023-02-13 12:22:47,129 INFO [Listener at localhost.localdomain/46179] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase16.apache.org,46677,1676290921946 2023-02-13 12:22:47,496 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 12:22:47,496 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 12:22:47,496 INFO [Listener at localhost.localdomain/46179] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 12:22:47,496 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 12:22:47,496 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 12:22:47,497 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 12:22:47,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 12:22:47,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 12:22:47,498 DEBUG [Listener at localhost.localdomain/46179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x006d038c to 127.0.0.1:62541 2023-02-13 12:22:47,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 12:22:47,499 DEBUG [Listener at localhost.localdomain/46179] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:47,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 12:22:47,499 INFO [Listener at localhost.localdomain/46179] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase16.apache.org,34139,1676290923061' ***** 2023-02-13 12:22:47,500 INFO [Listener at localhost.localdomain/46179] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 12:22:47,500 INFO [Listener at localhost.localdomain/46179] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase16.apache.org,34079,1676290923111' ***** 2023-02-13 12:22:47,500 INFO [Listener at localhost.localdomain/46179] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 12:22:47,500 INFO [Listener at localhost.localdomain/46179] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase16.apache.org,36049,1676290923148' ***** 2023-02-13 12:22:47,500 INFO [Listener at localhost.localdomain/46179] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 12:22:47,501 INFO [RS:0;jenkins-hbase16:34139] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 12:22:47,501 INFO [RS:1;jenkins-hbase16:34079] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 12:22:47,501 INFO [RS:2;jenkins-hbase16:36049] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 12:22:47,501 INFO [RS:0;jenkins-hbase16:34139] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 12:22:47,502 INFO [RS:2;jenkins-hbase16:36049] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 12:22:47,502 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 12:22:47,502 INFO [RS:0;jenkins-hbase16:34139] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 12:22:47,502 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 12:22:47,502 INFO [RS:0;jenkins-hbase16:34139] regionserver.HRegionServer(1145): stopping server jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:47,502 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 12:22:47,502 DEBUG [RS:0;jenkins-hbase16:34139] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5460a4d4 to 127.0.0.1:62541 2023-02-13 12:22:47,502 INFO [RS:1;jenkins-hbase16:34079] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 12:22:47,503 DEBUG [RS:0;jenkins-hbase16:34139] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:47,502 INFO [RS:2;jenkins-hbase16:36049] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 12:22:47,503 INFO [RS:0;jenkins-hbase16:34139] regionserver.HRegionServer(1171): stopping server jenkins-hbase16.apache.org,34139,1676290923061; all regions closed. 2023-02-13 12:22:47,503 INFO [RS:1;jenkins-hbase16:34079] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 12:22:47,503 INFO [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(3304): Received CLOSE for 1c79ba080d1eda25c45efcc8579418da 2023-02-13 12:22:47,503 INFO [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(1145): stopping server jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:47,504 DEBUG [RS:1;jenkins-hbase16:34079] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d1e48bf to 127.0.0.1:62541 2023-02-13 12:22:47,504 DEBUG [RS:1;jenkins-hbase16:34079] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:47,504 INFO [RS:1;jenkins-hbase16:34079] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 12:22:47,504 INFO [RS:1;jenkins-hbase16:34079] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 12:22:47,504 INFO [RS:1;jenkins-hbase16:34079] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 12:22:47,504 INFO [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(1145): stopping server jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:47,504 INFO [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 12:22:47,504 DEBUG [RS:2;jenkins-hbase16:36049] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32c50248 to 127.0.0.1:62541 2023-02-13 12:22:47,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 1c79ba080d1eda25c45efcc8579418da, disabling compactions & flushes 2023-02-13 12:22:47,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. 2023-02-13 12:22:47,504 DEBUG [RS:2;jenkins-hbase16:36049] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:47,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. 2023-02-13 12:22:47,504 INFO [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 12:22:47,504 INFO [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 12:22:47,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. after waiting 0 ms 2023-02-13 12:22:47,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. 2023-02-13 12:22:47,504 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 12:22:47,504 DEBUG [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-13 12:22:47,504 DEBUG [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(1479): Online Regions={1c79ba080d1eda25c45efcc8579418da=hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da.} 2023-02-13 12:22:47,504 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 12:22:47,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 1c79ba080d1eda25c45efcc8579418da 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-13 12:22:47,505 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 12:22:47,505 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 12:22:47,505 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 12:22:47,505 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.88 KB heapSize=212.75 KB 2023-02-13 12:22:47,506 DEBUG [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-13 12:22:47,506 DEBUG [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(1505): Waiting on 1c79ba080d1eda25c45efcc8579418da 2023-02-13 12:22:47,512 DEBUG [RS:0;jenkins-hbase16:34139] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/oldWALs 2023-02-13 12:22:47,512 INFO [RS:0;jenkins-hbase16:34139] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C34139%2C1676290923061:(num 1676290924738) 2023-02-13 12:22:47,512 DEBUG [RS:0;jenkins-hbase16:34139] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:47,512 INFO [RS:0;jenkins-hbase16:34139] regionserver.LeaseManager(133): Closed leases 2023-02-13 12:22:47,512 INFO [RS:0;jenkins-hbase16:34139] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 12:22:47,513 INFO [RS:0;jenkins-hbase16:34139] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 12:22:47,513 INFO [RS:0;jenkins-hbase16:34139] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 12:22:47,513 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 12:22:47,513 INFO [RS:0;jenkins-hbase16:34139] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 12:22:47,514 INFO [RS:0;jenkins-hbase16:34139] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:34139 2023-02-13 12:22:47,520 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=131.12 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/.tmp/info/24446d1401a849cd84080ebd2a2e5ac2 2023-02-13 12:22:47,523 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 24446d1401a849cd84080ebd2a2e5ac2 2023-02-13 12:22:47,537 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 12:22:47,538 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 12:22:47,537 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 12:22:47,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/namespace/1c79ba080d1eda25c45efcc8579418da/.tmp/info/ee3c30ce34704736a363f82c19dd2f6f 2023-02-13 12:22:47,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/namespace/1c79ba080d1eda25c45efcc8579418da/.tmp/info/ee3c30ce34704736a363f82c19dd2f6f as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/namespace/1c79ba080d1eda25c45efcc8579418da/info/ee3c30ce34704736a363f82c19dd2f6f 2023-02-13 12:22:47,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/namespace/1c79ba080d1eda25c45efcc8579418da/info/ee3c30ce34704736a363f82c19dd2f6f, entries=2, sequenceid=6, filesize=4.8 K 2023-02-13 12:22:47,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 1c79ba080d1eda25c45efcc8579418da in 51ms, sequenceid=6, compaction requested=false 2023-02-13 12:22:47,562 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/.tmp/rep_barrier/68a518a2e8e94164bd598128a46d64fa 2023-02-13 12:22:47,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/namespace/1c79ba080d1eda25c45efcc8579418da/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-13 12:22:47,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. 2023-02-13 12:22:47,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 1c79ba080d1eda25c45efcc8579418da: Waiting for close lock at 1676290967504Running coprocessor pre-close hooks at 1676290967504Disabling compacts and flushes for region at 1676290967504Disabling writes for close at 1676290967504Obtaining lock to block concurrent updates at 1676290967505 (+1 ms)Preparing flush snapshotting stores in 1c79ba080d1eda25c45efcc8579418da at 1676290967505Finished memstore snapshotting hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676290967505Flushing stores of hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. at 1676290967506 (+1 ms)Flushing 1c79ba080d1eda25c45efcc8579418da/info: creating writer at 1676290967506Flushing 1c79ba080d1eda25c45efcc8579418da/info: appending metadata at 1676290967510 (+4 ms)Flushing 1c79ba080d1eda25c45efcc8579418da/info: closing flushed file at 1676290967510Flushing 1c79ba080d1eda25c45efcc8579418da/info: reopening flushed file at 1676290967552 (+42 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 1c79ba080d1eda25c45efcc8579418da in 51ms, sequenceid=6, compaction requested=false at 1676290967555 (+3 ms)Writing region close event to WAL at 1676290967560 (+5 ms)Running coprocessor post-close hooks at 1676290967563 (+3 ms)Closed at 1676290967564 (+1 ms) 2023-02-13 12:22:47,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676290925165.1c79ba080d1eda25c45efcc8579418da. 2023-02-13 12:22:47,566 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 68a518a2e8e94164bd598128a46d64fa 2023-02-13 12:22:47,578 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 12:22:47,578 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/.tmp/table/4cfd8aae0a814f069c532990dde380e8 2023-02-13 12:22:47,578 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 12:22:47,581 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4cfd8aae0a814f069c532990dde380e8 2023-02-13 12:22:47,581 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/.tmp/info/24446d1401a849cd84080ebd2a2e5ac2 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/info/24446d1401a849cd84080ebd2a2e5ac2 2023-02-13 12:22:47,584 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 24446d1401a849cd84080ebd2a2e5ac2 2023-02-13 12:22:47,584 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/info/24446d1401a849cd84080ebd2a2e5ac2, entries=64, sequenceid=307, filesize=11.9 K 2023-02-13 12:22:47,585 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/.tmp/rep_barrier/68a518a2e8e94164bd598128a46d64fa as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/rep_barrier/68a518a2e8e94164bd598128a46d64fa 2023-02-13 12:22:47,587 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 68a518a2e8e94164bd598128a46d64fa 2023-02-13 12:22:47,587 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/rep_barrier/68a518a2e8e94164bd598128a46d64fa, entries=18, sequenceid=307, filesize=7.0 K 2023-02-13 12:22:47,588 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/.tmp/table/4cfd8aae0a814f069c532990dde380e8 as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/table/4cfd8aae0a814f069c532990dde380e8 2023-02-13 12:22:47,590 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4cfd8aae0a814f069c532990dde380e8 2023-02-13 12:22:47,590 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/table/4cfd8aae0a814f069c532990dde380e8, entries=22, sequenceid=307, filesize=7.0 K 2023-02-13 12:22:47,591 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.88 KB/141192, heapSize ~212.70 KB/217808, currentSize=0 B/0 for 1588230740 in 86ms, sequenceid=307, compaction requested=false 2023-02-13 12:22:47,595 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-13 12:22:47,595 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 12:22:47,596 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 12:22:47,596 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676290967504Running coprocessor pre-close hooks at 1676290967504Disabling compacts and flushes for region at 1676290967504Disabling writes for close at 1676290967505 (+1 ms)Obtaining lock to block concurrent updates at 1676290967505Preparing flush snapshotting stores in 1588230740 at 1676290967505Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=141192, getHeapSize=217808, getOffHeapSize=0, getCellsCount=997 at 1676290967505Flushing stores of hbase:meta,,1.1588230740 at 1676290967506 (+1 ms)Flushing 1588230740/info: creating writer at 1676290967506Flushing 1588230740/info: appending metadata at 1676290967512 (+6 ms)Flushing 1588230740/info: closing flushed file at 1676290967512Flushing 1588230740/rep_barrier: creating writer at 1676290967524 (+12 ms)Flushing 1588230740/rep_barrier: appending metadata at 1676290967542 (+18 ms)Flushing 1588230740/rep_barrier: closing flushed file at 1676290967542Flushing 1588230740/table: creating writer at 1676290967566 (+24 ms)Flushing 1588230740/table: appending metadata at 1676290967568 (+2 ms)Flushing 1588230740/table: closing flushed file at 1676290967568Flushing 1588230740/info: reopening flushed file at 1676290967582 (+14 ms)Flushing 1588230740/rep_barrier: reopening flushed file at 1676290967585 (+3 ms)Flushing 1588230740/table: reopening flushed file at 1676290967588 (+3 ms)Finished flush of dataSize ~137.88 KB/141192, heapSize ~212.70 KB/217808, currentSize=0 B/0 for 1588230740 in 86ms, sequenceid=307, compaction requested=false at 1676290967591 (+3 ms)Writing region close event to WAL at 1676290967593 (+2 ms)Running coprocessor post-close hooks at 1676290967595 (+2 ms)Closed at 1676290967596 (+1 ms) 2023-02-13 12:22:47,596 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 12:22:47,654 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 12:22:47,654 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 12:22:47,706 INFO [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(1171): stopping server jenkins-hbase16.apache.org,34079,1676290923111; all regions closed. 2023-02-13 12:22:47,706 INFO [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(1171): stopping server jenkins-hbase16.apache.org,36049,1676290923148; all regions closed. 2023-02-13 12:22:47,718 DEBUG [RS:2;jenkins-hbase16:36049] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/oldWALs 2023-02-13 12:22:47,718 DEBUG [RS:1;jenkins-hbase16:34079] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/oldWALs 2023-02-13 12:22:47,718 INFO [RS:2;jenkins-hbase16:36049] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C36049%2C1676290923148:(num 1676290924738) 2023-02-13 12:22:47,719 INFO [RS:1;jenkins-hbase16:34079] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C34079%2C1676290923111.meta:.meta(num 1676290924854) 2023-02-13 12:22:47,719 DEBUG [RS:2;jenkins-hbase16:36049] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:47,719 INFO [RS:2;jenkins-hbase16:36049] regionserver.LeaseManager(133): Closed leases 2023-02-13 12:22:47,719 INFO [RS:2;jenkins-hbase16:36049] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 12:22:47,720 INFO [RS:2;jenkins-hbase16:36049] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 12:22:47,720 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 12:22:47,720 INFO [RS:2;jenkins-hbase16:36049] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 12:22:47,720 INFO [RS:2;jenkins-hbase16:36049] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 12:22:47,721 INFO [RS:2;jenkins-hbase16:36049] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:36049 2023-02-13 12:22:47,725 DEBUG [RS:1;jenkins-hbase16:34079] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/oldWALs 2023-02-13 12:22:47,725 INFO [RS:1;jenkins-hbase16:34079] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C34079%2C1676290923111:(num 1676290924737) 2023-02-13 12:22:47,725 DEBUG [RS:1;jenkins-hbase16:34079] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:47,725 INFO [RS:1;jenkins-hbase16:34079] regionserver.LeaseManager(133): Closed leases 2023-02-13 12:22:47,725 INFO [RS:1;jenkins-hbase16:34079] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 12:22:47,726 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 12:22:47,726 INFO [RS:1;jenkins-hbase16:34079] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:34079 2023-02-13 12:22:47,900 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:47,900 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 12:22:47,900 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:47,900 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,34139,1676290923061 2023-02-13 12:22:47,900 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 12:22:47,900 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@13b96b27 rejected from java.util.concurrent.ThreadPoolExecutor@1786ed88[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:47,900 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 12:22:47,900 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 12:22:47,901 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3ba464f2 rejected from java.util.concurrent.ThreadPoolExecutor@1786ed88[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,064 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:48,065 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:48,065 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,36049,1676290923148 2023-02-13 12:22:48,065 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@77c8ecae rejected from java.util.concurrent.ThreadPoolExecutor@73f7668f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,066 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:48,065 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:48,066 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@37147541 rejected from java.util.concurrent.ThreadPoolExecutor@73f7668f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,065 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@43568ba4 rejected from java.util.concurrent.ThreadPoolExecutor@1786ed88[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,066 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@179abad2 rejected from java.util.concurrent.ThreadPoolExecutor@13a52cf9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,067 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,34079,1676290923111 2023-02-13 12:22:48,067 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2a30ff00 rejected from java.util.concurrent.ThreadPoolExecutor@1786ed88[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,129 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,34079,1676290923111] 2023-02-13 12:22:48,129 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,34079,1676290923111; numProcessing=1 2023-02-13 12:22:48,147 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,34079,1676290923111 already deleted, retry=false 2023-02-13 12:22:48,147 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase16.apache.org,34079,1676290923111 expired; onlineServers=2 2023-02-13 12:22:48,147 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,36049,1676290923148] 2023-02-13 12:22:48,148 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,36049,1676290923148; numProcessing=2 2023-02-13 12:22:48,156 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,36049,1676290923148 already deleted, retry=false 2023-02-13 12:22:48,156 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase16.apache.org,36049,1676290923148 expired; onlineServers=1 2023-02-13 12:22:48,156 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,34139,1676290923061] 2023-02-13 12:22:48,156 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,34139,1676290923061; numProcessing=3 2023-02-13 12:22:48,164 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,34139,1676290923061 already deleted, retry=false 2023-02-13 12:22:48,164 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase16.apache.org,34139,1676290923061 expired; onlineServers=0 2023-02-13 12:22:48,164 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase16.apache.org,46677,1676290921946' ***** 2023-02-13 12:22:48,164 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-13 12:22:48,165 DEBUG [M:0;jenkins-hbase16:46677] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68ad7754, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-02-13 12:22:48,165 INFO [M:0;jenkins-hbase16:46677] regionserver.HRegionServer(1145): stopping server jenkins-hbase16.apache.org,46677,1676290921946 2023-02-13 12:22:48,165 INFO [M:0;jenkins-hbase16:46677] regionserver.HRegionServer(1171): stopping server jenkins-hbase16.apache.org,46677,1676290921946; all regions closed. 2023-02-13 12:22:48,165 DEBUG [M:0;jenkins-hbase16:46677] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 12:22:48,165 DEBUG [M:0;jenkins-hbase16:46677] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 12:22:48,166 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 12:22:48,166 DEBUG [M:0;jenkins-hbase16:46677] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 12:22:48,166 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1676290924320] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1676290924320,5,FailOnTimeoutGroup] 2023-02-13 12:22:48,166 INFO [M:0;jenkins-hbase16:46677] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 12:22:48,166 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1676290924320] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1676290924320,5,FailOnTimeoutGroup] 2023-02-13 12:22:48,167 INFO [M:0;jenkins-hbase16:46677] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 12:22:48,167 INFO [M:0;jenkins-hbase16:46677] hbase.ChoreService(369): Chore service for: master/jenkins-hbase16:0 had [] on shutdown 2023-02-13 12:22:48,167 DEBUG [M:0;jenkins-hbase16:46677] master.HMaster(1502): Stopping service threads 2023-02-13 12:22:48,167 INFO [M:0;jenkins-hbase16:46677] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 12:22:48,167 ERROR [M:0;jenkins-hbase16:46677] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] Thread[HFileArchiver-9,5,PEWorkerGroup] Thread[HFileArchiver-10,5,PEWorkerGroup] 2023-02-13 12:22:48,168 INFO [M:0;jenkins-hbase16:46677] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 12:22:48,168 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 12:22:48,181 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 12:22:48,181 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 12:22:48,181 DEBUG [M:0;jenkins-hbase16:46677] zookeeper.ZKUtil(398): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 12:22:48,181 WARN [M:0;jenkins-hbase16:46677] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 12:22:48,181 INFO [M:0;jenkins-hbase16:46677] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 12:22:48,181 INFO [M:0;jenkins-hbase16:46677] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 12:22:48,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 12:22:48,181 DEBUG [M:0;jenkins-hbase16:46677] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 12:22:48,182 INFO [M:0;jenkins-hbase16:46677] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 12:22:48,182 DEBUG [M:0;jenkins-hbase16:46677] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 12:22:48,182 DEBUG [M:0;jenkins-hbase16:46677] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 12:22:48,182 DEBUG [M:0;jenkins-hbase16:46677] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 12:22:48,182 INFO [M:0;jenkins-hbase16:46677] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=844.34 KB heapSize=1.01 MB 2023-02-13 12:22:48,201 INFO [M:0;jenkins-hbase16:46677] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=844.34 KB at sequenceid=2037 (bloomFilter=true), to=hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2aaa73d5bb58483e93f3ab6a1e5f75ac 2023-02-13 12:22:48,203 INFO [M:0;jenkins-hbase16:46677] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2aaa73d5bb58483e93f3ab6a1e5f75ac 2023-02-13 12:22:48,204 DEBUG [M:0;jenkins-hbase16:46677] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2aaa73d5bb58483e93f3ab6a1e5f75ac as hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2aaa73d5bb58483e93f3ab6a1e5f75ac 2023-02-13 12:22:48,206 INFO [M:0;jenkins-hbase16:46677] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2aaa73d5bb58483e93f3ab6a1e5f75ac 2023-02-13 12:22:48,206 INFO [M:0;jenkins-hbase16:46677] regionserver.HStore(1080): Added hdfs://localhost.localdomain:43631/user/jenkins/test-data/80984146-6ea5-2373-3eae-fd1f614b0a83/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2aaa73d5bb58483e93f3ab6a1e5f75ac, entries=282, sequenceid=2037, filesize=30.7 K 2023-02-13 12:22:48,207 INFO [M:0;jenkins-hbase16:46677] regionserver.HRegion(2947): Finished flush of dataSize ~844.34 KB/864607, heapSize ~1.01 MB/1055544, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 25ms, sequenceid=2037, compaction requested=false 2023-02-13 12:22:48,208 INFO [M:0;jenkins-hbase16:46677] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 12:22:48,208 DEBUG [M:0;jenkins-hbase16:46677] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676290968181Disabling compacts and flushes for region at 1676290968181Disabling writes for close at 1676290968182 (+1 ms)Obtaining lock to block concurrent updates at 1676290968182Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676290968182Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=864607, getHeapSize=1055544, getOffHeapSize=0, getCellsCount=2524 at 1676290968182Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676290968183 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676290968183Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676290968191 (+8 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676290968191Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676290968204 (+13 ms)Finished flush of dataSize ~844.34 KB/864607, heapSize ~1.01 MB/1055544, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 25ms, sequenceid=2037, compaction requested=false at 1676290968207 (+3 ms)Writing region close event to WAL at 1676290968208 (+1 ms)Closed at 1676290968208 2023-02-13 12:22:48,211 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 12:22:48,211 INFO [M:0;jenkins-hbase16:46677] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 12:22:48,212 INFO [M:0;jenkins-hbase16:46677] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:46677 2023-02-13 12:22:48,222 DEBUG [M:0;jenkins-hbase16:46677] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase16.apache.org,46677,1676290921946 already deleted, retry=false 2023-02-13 12:22:48,301 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:22:48,301 INFO [RS:2;jenkins-hbase16:36049] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase16.apache.org,36049,1676290923148; zookeeper connection closed. 2023-02-13 12:22:48,301 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f829ff1 rejected from java.util.concurrent.ThreadPoolExecutor@73f7668f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,302 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:36049-0x10052d701450003, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:22:48,302 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4270c03d rejected from java.util.concurrent.ThreadPoolExecutor@73f7668f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,302 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@8dba9a8] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@8dba9a8 2023-02-13 12:22:48,401 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:22:48,402 INFO [RS:1;jenkins-hbase16:34079] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase16.apache.org,34079,1676290923111; zookeeper connection closed. 2023-02-13 12:22:48,402 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2535c4bb rejected from java.util.concurrent.ThreadPoolExecutor@13a52cf9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,402 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34079-0x10052d701450002, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:22:48,403 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3447a185] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3447a185 2023-02-13 12:22:48,403 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5d438c1f rejected from java.util.concurrent.ThreadPoolExecutor@13a52cf9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,502 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:22:48,502 INFO [RS:0;jenkins-hbase16:34139] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase16.apache.org,34139,1676290923061; zookeeper connection closed. 2023-02-13 12:22:48,502 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5852041e rejected from java.util.concurrent.ThreadPoolExecutor@1786ed88[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,503 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): regionserver:34139-0x10052d701450001, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:22:48,503 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@321e6ad0] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@321e6ad0 2023-02-13 12:22:48,503 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1b007162 rejected from java.util.concurrent.ThreadPoolExecutor@1786ed88[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,504 INFO [Listener at localhost.localdomain/46179] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-13 12:22:48,602 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:22:48,602 INFO [M:0;jenkins-hbase16:46677] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase16.apache.org,46677,1676290921946; zookeeper connection closed. 2023-02-13 12:22:48,603 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@52f95bc6 rejected from java.util.concurrent.ThreadPoolExecutor@64347ff6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 26] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,603 DEBUG [Listener at localhost.localdomain/46179-EventThread] zookeeper.ZKWatcher(600): master:46677-0x10052d701450000, quorum=127.0.0.1:62541, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 12:22:48,603 ERROR [Listener at localhost.localdomain/46179-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@20039a8b rejected from java.util.concurrent.ThreadPoolExecutor@64347ff6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 26] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 12:22:48,605 WARN [Listener at localhost.localdomain/46179] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 12:22:48,609 INFO [Listener at localhost.localdomain/46179] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 12:22:48,718 WARN [BP-1060943449-188.40.62.62-1676290918312 heartbeating to localhost.localdomain/127.0.0.1:43631] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 12:22:48,718 WARN [BP-1060943449-188.40.62.62-1676290918312 heartbeating to localhost.localdomain/127.0.0.1:43631] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1060943449-188.40.62.62-1676290918312 (Datanode Uuid 5b5fd4b5-8f31-4d5e-9179-89c46b38b1c7) service to localhost.localdomain/127.0.0.1:43631 2023-02-13 12:22:48,720 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/dfs/data/data5/current/BP-1060943449-188.40.62.62-1676290918312] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 12:22:48,720 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/dfs/data/data6/current/BP-1060943449-188.40.62.62-1676290918312] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 12:22:48,722 WARN [Listener at localhost.localdomain/46179] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 12:22:48,724 INFO [Listener at localhost.localdomain/46179] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 12:22:48,834 WARN [BP-1060943449-188.40.62.62-1676290918312 heartbeating to localhost.localdomain/127.0.0.1:43631] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 12:22:48,834 WARN [BP-1060943449-188.40.62.62-1676290918312 heartbeating to localhost.localdomain/127.0.0.1:43631] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1060943449-188.40.62.62-1676290918312 (Datanode Uuid b1c51ee3-df0b-49ef-bd7b-a0c7d8646013) service to localhost.localdomain/127.0.0.1:43631 2023-02-13 12:22:48,835 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/dfs/data/data3/current/BP-1060943449-188.40.62.62-1676290918312] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 12:22:48,836 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/dfs/data/data4/current/BP-1060943449-188.40.62.62-1676290918312] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 12:22:48,838 WARN [Listener at localhost.localdomain/46179] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 12:22:48,840 INFO [Listener at localhost.localdomain/46179] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 12:22:48,946 WARN [BP-1060943449-188.40.62.62-1676290918312 heartbeating to localhost.localdomain/127.0.0.1:43631] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 12:22:48,947 WARN [BP-1060943449-188.40.62.62-1676290918312 heartbeating to localhost.localdomain/127.0.0.1:43631] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1060943449-188.40.62.62-1676290918312 (Datanode Uuid 01bf2301-7426-448b-8288-7b5ccc31096b) service to localhost.localdomain/127.0.0.1:43631 2023-02-13 12:22:48,948 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/dfs/data/data1/current/BP-1060943449-188.40.62.62-1676290918312] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 12:22:48,949 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4bc3bcb0-d9a1-4b18-6f9f-406252608683/cluster_289ae387-9c4b-9bf7-ab26-5d8fbce30836/dfs/data/data2/current/BP-1060943449-188.40.62.62-1676290918312] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 12:22:48,979 INFO [Listener at localhost.localdomain/46179] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-02-13 12:22:49,118 INFO [Listener at localhost.localdomain/46179] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-13 12:22:49,203 INFO [Listener at localhost.localdomain/46179] hbase.HBaseTestingUtility(1293): Minicluster is down